Cloud

As an organization increasingly relies on cloud, its number of cloud projects can also increase. Over time, project sprawl creeps in, and an organization can be left with tens, or even hundreds, of unnecessary projects. While these projects can be deleted in bulk, it becomes challenging to determine which projects are no longer needed. As manual efforts to understand each project are undertaken, valuable resources go wasted performing this arduous task. Even worse, resources running in the superfluous projects could be increasing your costs, carbon footprint, and security risk.

Introducing Remora

Remora is a serverless solution that helps limit the number of unused projects in your organization. It works with the Unattended Project Recommender to notify project owners of their unused projects, escalate notifications, and then delete those projects if no action is taken after a predetermined period.

How it works

The way the solution works is straightforward. The unattended project recommender analyzes usage activity on projects in your organization to make recommendations about reclaiming or removing unattended projects. Taking those recommendations a step further, Remora was designed to identify owners of the unattended projects and then send them a customizable email notification or assign them a Jira ticket. You can establish a predefined cadence to send notifications, designating an Essential Contact to be copied after the first email (e.g., the folder owner). After three emails are sent for any given project, a time-to-live (TTL) can be set to determine how long the unused project can stay unused before it is removed. Remora labels each project with an impending deletion date.

Cloud Scheduler kicks off a Cloud Workflow that runs on a scheduled cadenceA Cloud Workflow will check for active or claimed unattended project recommendations for the organization via the Recommender APIA BigQuery table containing each recommendation and the number of times it has been processed is queried and updatedA Cloud Workflow will determine what actions need to be taken (e.g., deleting the project as well as which contacts should be notified) A Cloud Workflow will publish a message to a Pub/Sub topicA Pub/Sub subscription will push the message to a Cloud FunctionA Cloud Function will retrieve the Jira or Sendgrid credentials from Secret Manager in order to notify the designated contactsJira or Sendgrid will notify project contacts via a ticket or email, respectively, regarding the unattended projectsThe project contacts will be able to act on the unattended project recommendations, either by applying or dismissing themApplying or dismissing the recommendation will update its state on the Recommender API

Remora’s core capabilities

Remora was built with several essential capabilities to ensure it could be customized to help meet each organization’s unique requirements:

Dry-run mode:dry-run mode is enabled by default, which prevents Remora from deleting projects. Dry-run mode must be turned off in order for projects to be deleted by the solution.

Multiple notifications:owners of unused projects should have multiple opportunities to act on the recommendations. Remora notifies owners every time it runs, and Cloud Scheduler can be used to set up periodic Remora runs (e.g., once a week).

Summary notifications:an owner on multiple unused projects receives a single email notification with all the projects identified.

Escalation of notifications:the first notification is always sent directly to the project owner(s). We’ve implemented two mechanisms for escalations of subsequent notifications:

Essential Contacts: Remora escalates to the specified category of Essential Contacts for the project. If your identities are different from your email addresses, configuring Essential Contacts will inform Remora of the correct escalation email addresses.

Folder or organization admins: when an Essential Contacts category is not specified, Remora escalates to the admin of the project’s parent folder or organization (whichever is the parent in the resource hierarchy).

Time-to-live:Organization admins can set the number of days during which an unused project can remain in their organization. Remora will label the projects with their impending deletion date and delete the projects after the designated period of time and three notifications.

Notification mechanisms:Remora sends email notifications using Sendgrid or creates Jira tickets. 

Deployment using Google Cloud CLI or Terraform: Remora can be deployed manually using gcloud commands or as a Terraform module.

The holistic solution

The entire solution is made up by combining the components below. 

Unattended Project Recommender

The unattended project recommender analyzes project usage and provides recommendations to remove unused projects. Generally, a project will be recommended for deletion when it has low usage for 30 days and no OAuth tokens used in the last 180 days. Remora will then label the unattended project for deletion.

Google Cloud Workflows and Scheduler 

Workflows is a service that lets you connect different Google Cloud services and APIs to create pipelines and process automation. Workflows are configured with a YAML or JSON file that lists a series of steps in their order of execution. For this solution, Workflows are used to create the initial BigQuery dataset and tables where recommendations will be tracked, retrieve the latest unattended project recommendations from the Recommender API, and call Pub/Sub to initiate the notification process to the owners of the identified unattended projects. The workflows execute on a schedule configured using Cloud Scheduler, Google Cloud’s crontab-as-a-service solution. Cloud Scheduler is where you configure how often you want Remora to process unattended project recommendations.

Cloud Functions

Cloud Functions is Google Cloud’s function-as-a-service offering that lets you execute lightweight functions without the need to manage any servers. Cloud Functions can execute programmatically when triggered by events from Cloud Storage, Pub/Sub, Firebase or HTTP requests. Here, a Cloud Function is triggered via Pub/Sub to alert the project owner via email using Sendgrid or via an issue in Jira. 

Terraform

To simplify and streamline the deployment of Remora, we compiled the individual Google Cloud CLI commands into a Terraform module that creates all the resources needed to get Remora running. As a Terraform module, Remora can be deployed to the provided Google Cloud project and customized with just a few variables. 

The module will handle the creation and configuration of Workflows, Cloud Scheduler, Cloud Functions, and a service account with custom role assignments on the project and organization IAM policy. The code used for the Cloud Functions is included in the module and is uploaded as an archive file to a Cloud Storage bucket.

Check out the documentation in the repository for more detailed usage information and examples. Here’s one simple example of what a module might look like:

The example above will retrieve recommendations from the unattended project recommender every Sunday night, then use Sendgrid to send an email to the unattended project owner. 

Putting it into practice

As soon as Remora is deployed in your organization, Workflows will query the Recommender API based on a specified interval. With Sendgrid configured as the notifier, the project owner will receive a message like this:

After being notified, the unattended project owner will have two options: delete the project right away, or dismiss the recommendation so that it won’t be picked up by the Recommender API again. 

If no action is taken after the first notification, the next notification will include your specified category of Essential Contacts. If the Essential Contacts category is not set, the next owner in the resource hierarchy (i.e., the folder or organization) is included instead. The second message will look like this:

Finally, if no action is taken after three notifications and the TTL has expired, the project is automatically shut down and marked for deletion when Remora runs. Just like shutting down a project manually, there is a 30-day period where the project can be restored in case it was deleted in error. 

By leveraging the intelligence of Active Assist’s recommendation APIs, Workflows, and Cloud Functions, Remora will prune unattended projects to potentially resolve security risks, reduce your carbon footprint, and lower the associated costs of your cloud infrastructure without the overhead incurred from frequent manual auditing. Additionally, since Remora is an open-source project, you can examine and customize the logic used in the Workflows and Cloud Functions to tailor the solution to your organization’s needs. You can get started by checking out the project repository on GitHub and deploying Remora using the provided Terraform module. If you would like to learn more about Active Assist, please take a look at this YouTube playlist covering Active Assist and its intelligent features.

One benefit of cloud migration is having access to managed services, which can reduce operational overhead. For organizations operating in Microsoft-centered environments, Google Cloud offers a highly-available, hardened Managed Service for Microsoft Active Directory running on Windows virtual machines. Managed Microsoft AD provides benefits such as automated AD server updates, maintenance, default security configurations, and needs no hardware management or patching. 

Most organizations adopting Managed Microsoft AD will be migrating from an existing on-premises AD deployment. Typically, when migrating Active Directory objects, existing users cannot continue to access resources in the new domain unless security identifier (SID) history is preserved. This can lead to additional work for administrators as the permissions need to be recreated post migration.

To make migrations more seamless and eliminate extra effort, we are excited to announce a new capability in Managed Microsoft AD: support for the migration of AD users with SID history, now available in Preview. Now, users can retain historic Access Control List (ACL) entries so that users can access resources without having to recreate resource permissions post-migration. 

Steps to migrate on-premises AD users to Managed Microsoft AD

To get started, you can use Active Directory Migration Tool (ADMT) to migrate an on-premises AD domain to Managed Microsoft AD specifically with SID history. 

1. Prepare your on-premises Active Directory and Managed Microsoft AD

As a prerequisite for migration, users need to set up a two-way trust between existing on-premises AD domain and new Managed Microsoft AD domain. 

Either a single user or a team within the users’ organization can perform the migration activities. When a team is involved, we recommend adding the team members to a domain local group in Managed Microsoft AD. Users can connect to a Managed Microsoft AD domain and use the standard Active Directory tools such as Active Directory Users and Computers (ADUC) that is part of RSAT: Active Directory Domain Services for adding those users to the domain local group. Remember that users need to add this domain local group to the pre-created groups in Managed Microsoft AD, after enabling permissions as described in step 3.

2. Prepare a Google Compute Engine Virtual Machine and set up ADMT

As a next step, install and set up Microsoft Active Directory Migration Tool (ADMT) and Microsoft SQL Server 2016 Express on a Google Compute Engine Virtual Machine. You need to make sure that this VM is not a domain controller and then join the VM to Google Cloud Managed Microsoft AD domain.

3. Enable permissions on Managed Microsoft AD

After users prepare the on-premises Active Directory and Managed Microsoft AD, they can enable the required permissions in Managed Microsoft AD to migrate the users with SID History. You can use the following gCloud CLI command:

After users enable the permissions, add the domain local group to the “Cloud Service Administrators” and “Cloud Service Migrate SID Administrators” (these groups are pre-created in Managed Microsoft AD) so that the designated users will have the required permissions for Active Directory migration with SID history.

For more information, see Enable permissions for migrating an on-premises domain with SID history and also ensure you understand these security implications. 

4. Configure ADMT and install PES service

To migrate passwords securely during the domain migration, users need to use the Microsoft Password Export Server (PES) service. Before users install PES on the on-premises domain, create the encryption key on the VM running ADMT. Users can use the following command to create the encryption key:

It creates and stores the encryption key at the location specified at KEY_FILE_PATH. You can copy the encryption key to the local drive of the on-premises domain controller where you have installed the PES service. 

Note: It is recommended to run the PES service only when you migrate passwords.

5. Migrate AD users to Google Cloud Managed Microsoft AD

Now that users have the environment ready and the required permissions enabled, migrate the AD users & groups along with the SID history. Users will have to use the User Account Migration Wizard in the ADMT tool. For detailed steps, please refer to the ADMT guide. In this wizard,  select the “Migrate user SIDs to target domain” option on the Account Transition Options page.

Note: You may want to migrate other AD objects as well, so as a best practice, we recommend that you maintain a migration checklist and verify managed objects in the checklist post-migration.

The migration process window will show the status of how many users, groups and computers have been migrated. You can click on “View Log” to examine the migration logs for errors and check the details of the operations.

After the migration process is completed, verify that the users exist in the Managed Microsoft AD domain using the Active Directory Users and Computers tool. Users can also use the AD login to verify that the access is set up correctly and verify that users can login successfully with resource permissions based on the SID history. 

Now users have successfully migrated AD objects including users along with SID history from  on-premises AD domain to a new Managed Microsoft AD domain. After successful migration, we recommend disabling the user permissions that was enabled for migration.

Next Steps

We are continually evolving our Managed Microsoft AD features to help you more easily and effectively manage AD tasks. Here are additional resources where you can learn more about Managed Microsoft AD and these new features.

Managed Service for Microsoft AD documentationExisting domain migration overviewEnable permissions for migrating an on-premises domain with SID history

Every year, government agencies are responsible for distributing services and benefits that require processing billions of documents, images and forms. These documents are often expected to be manually transcribed into an electronic system error-free by a government employee to support time-sensitive tasks such as application intake, verification, enrollment, vendor management, and procurement. 

While agencies want to process these documents quickly to support their citizens, they’re often challenged with the following: 

1. Most applications require a variety of custom forms and verification documents (e.g. tax forms and utility bills) that are received at different times and in different formats (paper, images, etc.), making it difficult to track and match documents to a single applicant. 

2. Handwritten entries and those in multiple languages add to the challenges of understanding and entering data quickly.

3. Document processing is labor intensive when scanning, reading, data entry, indexing, and matching are all done manually

 

With Document AI, state, local and federal agencies can accelerate time to delivery of critical services by decreasing the manual labor needed to process documents. 

This solution can assist government workers by automatically extracting content from unstructured and handwritten documents and keying data into their existing system of record. We have expanded the capability to understand documents more easily and accurately over the years. As such, we are excited to announce new functionality that addresses the unique challenges faced by public sector agencies and their employees.

Introducing pre-built Document AI models to support common government use cases

Document AI for government includes pre-built document models for many widely used government document types, including invoices, receipts, driver’s license, passport, payslips, tax forms, and more. There may be scenarios where you want to extract additional fields or improve accuracy with your own documents. We now offer the ability to transfer learning or uptrain a pre-built model.

 

For scenarios where constituents are dealing with custom documents where a pre-built model may not exist, we offer Document AI Workbench.  With the release of Document AI Workbench, government agencies can now create custom document models to extract fields from any document, image and form, enabling you to accelerate and streamline almost any workflow that requires heavy document processing. 

Finally, with Document AI Accelerator,  government workers can now use a lightweight workflow tool to manage the flow of incoming applications by identifying and classifying  documents, matching documents to a single constituent and proactively reaching out if there is missing or inaccurate information. We also provide the ability to integrate human reviews for accuracy and validation.

Document AI helps unlock insights for a variety of use cases at Government agencies. Be it extracting fields from custom documents, or specific documents such as driver’s license, passport, taxes, or text in over 200 languages, Document AI offers a way more easily to process and automate documents. 

The State of Hawaii used Document AI to extract travel and health information from visitors to safely reopen to tourism. “You need technology that can scale up easily and continue to be fast. The system now routinely handles 25,000 or more per day..” Doug Murdock, CIO, Office of Enterprise Technology Services, State of Hawaii.

Learn more about how the government is accelerating innovation at ourGoogle Government Summit and upcoming coffee hour on Automated Document Processing.