Cloud

Introduction

When it comes to cloud-based content management, collaboration, and file sharing tools for businesses and individuals, Box, Inc. is a recognized leader. Recently, we decided to migrate from Apache HBase, a distributed, scalable, big data store deployed on-premises, to Cloud Bigtable, Google Cloud’s HBase-compatible NoSQL database. By doing so, we achieved the many benefits of a cloud-managed database: reduced operational maintenance work on HBase, flexible scaling, decreased costs and an 85% smaller storage footprint. At the same time, the move allowed us to enable BigQuery, Google Cloud’s enterprise data warehouse, and run our database across multiple geographical regions.

But how? Adopting Cloud Bigtable meant migrating one of Box’s most critical services, whose secure file upload and download functionality is core to its content cloud. It also meant migrating over 600 TB of data with zero downtime. Read on to learn how we did it, and the benefits we’re ultimately enjoying. 

Background

Historically, Box has used HBase to store customer file metadata with the schema in the table below. This provides us a mapping from a file to a file’s physical storage locations. This metadata is managed by a service called Storage Service, which runs on Kubernetes; this metadata is used on every upload and download at Box. For some context on our scale: at the start of the migration we had multiple HBase clusters that each stored over 600 billion rows and 200 terabytes of data. Additionally, these clusters received around 15,000 writes per second and 20,000 reads per second, but could scale to serve millions of requests for analytical jobs or higher loads.

Our HBase architecture consisted of three fully replicated clusters spread across different geographical regions: Two active clusters for high availability, and another to handle routine maintenance. Each regional Storage Service wrote to its local HBase cluster and those modifications were replicated out to other regions. On reads, Storage Service first fetched from the local HBase cluster and fell back onto other clusters if there was a replication delay.

Preparing to migrate

To choose the best Bigtable cluster configuration for our use case, we ran performance tests and asynchronous reads and writes before the migration. You can learn more about this on the Box blog here. 

Since Bigtable requires no maintenance downtime, we decided to merge our three HBase clusters down to just two Bigtable clusters in separate regions for disaster recovery. That was a big benefit, but now we needed to figure out the best way to merge three replicas into two replicas!

Theoretically, the metadata in all three of our HBase clusters should have been the same because of partitioned writes and guaranteed replication. However, in practice, metadata across all the clusters had drifted, and Box’s Storage Service handled these inconsistencies upon read. Thus, during the backfill phase of the migration, we decided to take snapshots of each HBase cluster and import them into Bigtable. But we were unsure about whether to overlay the snapshots or to import the snapshots to separate clusters.

To decide on how to merge three clusters to two, we ran the Google-provided Multiverse Scan Job, a customized MapReduce job that sequentially scans HBase table snapshots in parallel. This allowed us to effectively perform a ​​sort-merge-join of the three tables and compare rows and cells for differences between the three HBase clusters. While the job scanned the entire table, a random 10% of critical rows were compared. This job took 160 Dataproc worker nodes and ran for four days. Then, we imported the differences into BigQuery for analysis.

We found that inconsistencies fell into three categories:

Missing rows in an HBase cluster

A row existed, but was missing columns in an HBase cluster

A row existed, but had differing non-critical columns in an HBase cluster

This exercise helped us decide that consolidating all three snapshots into one would provide us with the most consistent copy, and to have Bigtable replication handle importing the data into the secondary Bigtable cluster. This would resolve any issues with missing columns or rows.

Migration plan

So, how do you migrate trillions of rows into a live database? Based on our previous experience migrating a smaller database into Bigtable, we decided to implement synchronous modifications. In other words, every successful HBase modification would result in the same Bigtable modification. If either step failed, the overall request would be considered a failure, guaranteeing atomicity. For example, when a write to HBase succeeded, we would issue a write to Bigtable, serializing the operations. This increased the total latency of writes to the sum of a write to HBase and Bigtable. However, we determined that was an acceptable tradeoff, as doing parallel writes to both these databases would have introduced complex logic in Box’s Storage Service.

One complexity was that Box’s Storage Service performed many check-and-modify operations. These couldn’t be mirrored in Bigtable for the duration of migration where Bigtable had not been backfilled, and consequently check-and-modify operations would differ from the HBase check-and-modifies. For this reason, we decided to rely on the result of the HBase check-and-modify, and would only perform the modification if the HBase check-and-modify succeeded.

Rollout plan

To roll out synchronous modifications safely, we needed to control it by both percentage and region. For example, our rollout plan for a region looked like the following:

1% region 15% region 125% region 150% region 1100% region 1

Synchronous modifications ensured that Bigtable had all new data written to it. However, we still needed to backfill the old data. After running synchronous modifications for a week and observing no instabilities, we were ready to take the three HBase snapshots and move onto the import phase.

Bigtable import: Backfilling data

We had three HBase snapshots of 200TB each. We needed to import these into Bigtable using the Google-provided Dataproc Import Job. This job had to be run carefully since we were fully dependent on the performance of the Bigtable cluster. If we overloaded our Bigtable cluster, we would immediately see adverse customer impact — an increase in user traffic latency. In fact, our snapshots were so large that we scaled up our Bigtable cluster to 500 nodes to avoid any performance issues. We then began to import each snapshot sequentially. An import of this size was completely unknown to us so we controlled the rate of import by slowly increasing the size of the Dataproc cluster and monitoring Bigtable user traffic latencies.

Validation

Before we could start relying on reads from Bigtable, there was a sequence of validations that had to happen. If any row was incorrect, this could lead to negative customer impact. The size of our clusters made it impossible to do validation on every single row. Instead we took three separate approaches to validation to gain confidence on the migration:

1. Async read validation: Optimistic customer-download-driven validation

On every read, we asynchronously read from Bigtable and added logging and metrics to notify us of any differences. The one caveat with this approach was that we have a lot of reads that are immediately followed by an update. This approach created a lot of noise, since all of the differences we surfaced were from modifications that happened in between the HBase read and the Bigtable read.

During this read validation we discovered that Bigtable regex scans were different from HBase regex scans. For one, Bigtable only supports “equals” regex comparators. Also, the Bigtable Regex uses RE2 which treats “.” (any character, which unless specified excludes newline) differently than HBase. Thus, we had to roll out a specific regex for Bigtable scans and validate that they were returning the expected results.

2. Sync validation: Run a Dataproc job with hash comparison between Bigtable and HBase

This validation job, similar to the one found here, performed a comparison of hashes across Bigtable and HBase rows. We ran it on a sample of 3% of the rows and uncovered a 0.1% mismatch. We printed these mismatches and analyzed them. Most of these mismatches were from optimistic modifications to certain columns and found that no re-import or correction was needed.

3. Customer perspective validation 

We wanted to perform an application-level validation to see what customers would be seeing instead of a database-level validation.

We wrote a job to scan the whole filesystem that would queue up objects where we would call an endpoint in Storage Service that would compare the entry in Bigtable and HBase. For more information, check out this Box blog.

This validation supported the output of the Sync Validation job. We didn’t find any differences that weren’t explained above.

Flipping to Bigtable

All these validations gave us the confidence to return reads from Bigtable instead of HBase. We kept synchronous dual modifications to HBase on as a backup, in case we needed to roll anything back. After returning only Bigtable data, we were finally ready to turn off modifications to HBase. At this point Bigtable became our source of truth.

Thumbs up to Bigtable

Since completing the migration to Bigtable, here are some benefits we’ve observed. 

Speed of development

We now have full control of scaling up and down Bigtable clusters. We turned on Bigtable autoscaling, which automatically increases or decreases our clusters given CPU and storage utilization parameters. We were never able to do this before with physical hardware. This has allowed our team to develop quickly without impacting our customers.

Our team now has much less overhead related to managing our database. In the past, we would constantly have to move around HBase traffic to perform security patches. Now, we don’t need to worry about managing that at all.

Finally, MapReduce jobs that would take days in the past now finish under 24 hours.

Cost savings

Before Bigtable, we were running three fully replicated clusters. With Bigtable, we are able to run one primary cluster that takes in all the requests, and one replicated secondary cluster that we could use if there were any issues with the primary cluster. Besides, for disaster recovery, the secondary cluster is extremely useful to our team to run data analysis jobs.

Then, with autoscaling, we can run our secondary cluster much more lightly until we need to run a job, at which point it self-scales. The secondary cluster runs with 25% less nodes than the primary cluster. When we used HBase, all three of our clusters were sized evenly.

New analysis tools

We ported all our HBase MapReduce jobs over to Bigtable, and found that Bigtable has provided us with parity in functionality with minor configuration changes to our existing jobs.

Bigtable has also enabled us to use the Google Cloud ecosystem:

We were able to add Bigtable as an external BigQuery source. This allowed us to query our tables in real time, which was never possible in HBase. This application was best suited to our small tables. Care should be taken with running queries on a production Bigtable cluster due to impact on CPU utilization. App profiles may be used to isolate traffic to secondary clusters.

For our larger tables we decided to import them into BigQuery through a Dataproc job. This enabled us to pull ad hoc analytics data without running any extra jobs. Further, querying BigQuery is also much faster than running MapReduce jobs.

Long story short, migrating to Bigtable was a big job, but with all the benefits we’ve gained, we’re very glad we did! 

Considering a move to Bigtable? Find more information about migrations and Google Cloud supported tools:

Learn about our live migration tools such as HBase Replication and the HBase mirroring client

Walk through the migration guide for step-by-step instructions

Watch Box’s presentation, How Box modernized their NoSQL databases with minimal effort and downtime

Knowledge graphs are inevitably built with incomplete information, complete knowledge being somewhat hard to come by. Knowledge graphs stored in Neo4j Graph Database can be enriched with data from Google Enterprise Knowledge Graph to provide more complete information. 

Knowledge graphs enriched in this way contain more information, enabling the systems that use them to answer more questions. Those downstream systems can then make more accurate business decisions and provide better customer experiences.

In this blog post, we’ll demonstrate how to use Neo4j in conjunction with Google Enterprise Knowledge Graph to enrich a knowledge graph.

Neo4j

Neo4j Aura is the leading graph database. It is available on Google Cloud Marketplace. Neo4j Aura is the only graph database embedded in the Google Cloud Console. Neo4j AuraDS (Data Science) includes three components:

Neo4j Graph Database (GDB) – Graph database for storing, retrieving and managing data natively as graphs

Neo4j Graph Data Science (GDS) – A collection of 60+ graph algorithms for computing centrality, embeddings, etc

Neo4j Bloom – A graph specific business intelligence (BI) tool

Google Enterprise Knowledge Graph

Google Enterprise Knowledge Graph consists of two services: 

The Entity Reconciliation API lets customers build their own private Knowledge Graph with data stored in BigQuery. 

Google Knowledge Graph Search API lets customers search for more information about their entities, such as official name, types, description, etc. 

Architecture

The easiest way to enrich a knowledge graph stored in Neo4j Aura with Google Enterprise Knowledge Graph is to access both tools via Python APIs from Vertex AI Workbench. The architecture for such a system is shown below

Example with Private Equity Data

To illustrate this architecture, we’re going to look at some data from ApeVue, a data provider for private equity markets information. ApeVue aggregates price data on pre-IPO companies from a variety of private equity brokers and distributes market consensus prices and index benchmarks based on that information.

For this example we’re going to construct a knowledge graph in Neo4j Aura from some CSV files provided by ApeVue. These files cover the ApeVue50 Benchmark Index and its constituent companies. An iPython notebook that runs the load is available here. You’re more than welcome to run it yourself!

When complete, we can inspect the graph and see the index and subindices. One way to do that graphically is with Neo4j Browser. You can open that from within the Neo4j Aura console.

Now, we run the query: 

MATCH (n:Sector) RETURN n

The result is a view of all the sectors in the data set, in this case 14 of them.

We can click on any of those sectors to view the companies which are part of it as well:

Similarly, we can search for specific items within our knowledge graph and then view the neighborhood around them. For instance, we can search for Neo4j by running the Cypher query: 

MATCH (n:Company {name: ‘Neo4j’}) return n

When complete, expanding the neighborhood around the Neo4j node gives an interesting view of how it relates to other firms and investors:

In this case, we can see that Neo4j has a number of investors (the blue nodes). One of these is named “GV Management Company, LLC,” previously known as Google Ventures.

Enriching the Data with Enterprise Knowledge Graph

Our knowledge graph is pretty interesting in and of itself. But we can make it better. There’s no way to view the graph underlying Enterprise Knowledge Graph. However, we can query that graph and use it to enrich the knowledge graph we’ve built in Neo4j. In this case, we’re going to pass the company name into the API and get the description of that company back. We’ll then add a new property called “description” to each “Company” node in the Neo4j database.

That’s done via some Python code running in Vertex AI Workbench. A notebook showing an example of this is available here. The main part of that is the call to the Cloud Knowledge Graph API here:

Once we’ve run through the notebook, it’s possible to inspect the enriched data set in Neo4j. For instance, if we query for “Neo4j” again in our knowledge graph, we’ll see it now has a property called description:

What we’ve done here is take the best of both worlds. We’ve leveraged Google’s incredible expertise in search and knowledge aggregation with Neo4j’s market leading ability to store, query and visually represent data as graphs. The result is a very powerful way to build custom knowledge graphs.

Exploring the Knowledge Graph

Now that we’ve loaded and enriched our data, let’s explore it. To do so we’ll use Neo4j Bloom, a graph specific business intelligence tool.

First off, let’s take a look at the companies and graphs:

Here we can see our 14 sectors as the orange dots. A few sectors only have one company as part of them but others have many. Rerunning the search to include our investors, we now get a much more crowded view:

We can use the Neo4j Graph Data Science component to better make sense of this picture. Computing the betweenness centrality on the graph gives us this view.

In it we’ve sized nodes according to their score. One surprising thing in this is that the investors aren’t what’s important. Rather, the companies and sectors are the main connecting points in our graph.

We can also add filters to our perspective. For instance, we can filter to see what companies had a return of 10% or greater in the last month. That gives us this view:

Zooming in on that view, we can identify a firm that invested in four of those companies that returned 10% or greater.

This is a small subset of what’s possible with Bloom. We invite you to explore this data on your own!

Conclusion

In this blog post we loaded a dataset provided by ApeVue into Neo4j AuraDS. AuraDS is Neo4j’s managed service, running on Google Cloud Platform. The data we loaded consisted of the ApeVue50 index, including return, depth and open interest data. We queried the data using Neo4j Cypher and Browser. We enriched that dataset with queries to Google Enterprise Knowledge Graph. Finally, we explored the dataset using Neo4j Bloom.

The enriched knowledge graph gives us a more complete view of the private equity data. Downstream systems using this knowledge graph would have access, not just to return investor and sector information, but also to a natural language description of what the company does. This could be used to power conversational agents or be fed into full text searches of the data set. 

Systems that use Neo4j and Google Enterprise Knowledge Graph can provide better insights and understanding of connected information than otherwise possible. Downstream systems can then make more accurate business decisions and provide better customer experiences.

You can find the notebooks we used on GitHub here.

If you’d like to learn more about the dataset, please reach out to Nick Fusco and the team at ApeVue (contact@apevue.com). For information on Neo4j, please reach out to ecosystem@neo4j.com with any questions.

We thank the many Google Cloud and Neo4j team members who contributed to this collaboration.

Editor’s note: March is celebrated as Women’s History Month in the US. At Google, we’re proud to celebrate women from all backgrounds and are committed to increasing the number of women in the technology industry. Over the next few weeks, we will showcase women-led startups and how they use Google Cloud to grow their businesses.Today’s feature highlights E-learning platform, Schoolio, and its co-founder, Lindsey Casselman. 

I like to say that my favorite thing about Schoolio is that it all started with an act of kindness- and a cold DM. 

I began my career as a teacher, with the eager hope of most new teachers that I could make a difference for kids. A few years later, with two kids of my own in school, I felt that I couldn’t change the system from within, and saw it as antiquated, and at best leaving kids ill-prepared for the real world and the future of work. I left teaching, and in 2015 I began homeschooling my children, and never looked back. As a homeschooler, I immediately discovered that once you leave the system, you’re on your own. There is very little homeschool content out there, and even less of it is secular, inclusive, and diversity driven. I also learned that many homeschooled children are kids with unique needs, as they are the most poorly served in a traditional classroom. 

I used my knowledge of education and experience with children with unique needs to create home learning content for my own kids, and I became a leader in the community, mentoring new homeschoolers through a Facebook support group I started, where I would often share tips and some of my homemade teaching materials.

Everything changed in March 2020 when the COVID-19 pandemic struck. With millions of children suddenly learning at home, the number of people in my Facebook group exploded, and many were asking me to share more of my homemade content with them, as they became starkly aware of the lack of appropriate materials on the market.. It was great to provide support, but I knew I could do much more than just share screenshots of my handwritten lesson plans. Luckily the group also contained my co-founder, Sathish Bala, who reached out to me in what’s now a framed-in-the-office cold DM, asking if I was interested in collaborating. Within a few weeks, in August 2020, we launched our first set of E-books for home education, andSchoolio was born.

Sathish and I share a common approach to education. Schoolio is founded on our belief that schools must be reimagined for the modern world. We believe that education should prepare children for their real futures, and that interest-based learning is what truly keeps students engaged and empowered. This is how they will grow to love learning, realize their own potential, and be set up for lifelong success. 

After a successful seed fundraise in 2021, we have been able to completely digitize our content library into video format, and now offer the options of pen-to-paper book learning from the e-book store, or easy online access through our learning platform to meet the needs of families across the globe.  Our curriculum now includes K8 core subjects (math, language, science, and social studies) for Canada and the U.S, plus a library of interest-based topics, including climate science, a robust Financial Literacy program, and a complete SEL (Social-emotional learning) library, my personal passion-project, that helps kids develop social skills, emotional regulation strategies, and character-building tools- needed now more than ever in our post-Covid world. 

At the end of 2022 we also launched an immersive learning environment that takes children into a safe, curated 3D digital world. There’s really no limit to the educational experiences potentially available in this metaverse environment, from going on a virtual field trip to meeting teachers and experts from all over the world.

In addition, we are dedicated to using the latest innovations in technology to go beyond academic assessments and help children discover their passions and talents, learning styles and habits, strengths, and emotional well-being, as part of a whole-child approach to education. We believe there’s so much more to success than math scores!

Acceleration for education

I’m still amazed that we’ve achieved so much in two busy years. A lot of that is thanks to the friends and business partners that we’ve met along the way, and theGoogle Cloud team definitely belongs in both categories.

I’ve always admired Google Cloud for its innovative technology and as a force for good especially with non-profits, social enterprises, and education. We participated in the Google Cloud Accelerator Canada program, a 10-week virtual course that we joined in 2022. Along with cloud mentorship and technical project support, we also gained valuable insights into product design, customer acquisition, and leadership development. 

Our specific program was an accelerator for startups, which is a great description of the experience-we felt literally accelerated by the many helpful and interesting people who gave us a strong push in the right direction and helped us maintain momentum after the program.  

We were also eligible for $100,000 of Google Cloud credits through the Google for Startups Cloud Program, which gave us a financial boost. The Google Cloud team also introduced us to its objectives and key results (OKR) process which helped us articulate our goals and measure progress, and which we continue to use in-house today. 

From a technology point of view, we have deployedApp Engine as our cloud computing platform for developing and hosting web applications. We’re planning to deployBigQuery as our data warehouse andBigQuery ML which simplifies the creation of machine learning models. We’re also looking forward to the integration of more artificial intelligence features that will further streamline and accelerate our holistic education goals.

Breaking out of comfort zones

During the Google Cloud Accelerator Canada program, I was inspired by the other women tech founders that I met. I’ve been to events where I’m the only woman in the room,  or where stereotypes about women in tech are present, so when you meet people with a similar experience it helps you see  that you’re not alone and are actually part of a team of powerful innovators pushing boundaries for others. The Google team even connected me with members of the Google for Startups Accelerator: Women Founders cohort even though I wasn’t officially a member.

My advice to other women founders is to stay inspired to do what got you started in the first place, and don’t ever let the business or tech world make you feel like you don’t belong or like you need to change who you are in order to deserve to be there. Don’t be afraid to get out of your comfort zone. I was a teacher and then a stay-at-home mom, which goes to show that you don’t need years of business experience to be a pioneer. You can join the game whenever you want, as long as you have the passion and a willingness to continually learn new things. If you have initiative and you’re a self-learner, you are unstoppable.

The best of the journey is when you can look back at how you overcame your challenges and all that you’ve achieved. Projects that seemed insurmountable at the start eventually get smaller, and then completed, and before you know it you’re looking for the next mountain to climb. 

We’re looking forward to working with the Google Cloud team again in the future. When people ask me why we choose Google over others, I tell them that we get the best of both worlds: groundbreaking tech that opens the door to new education possibilities, and a fail-safe suite of applications that never let you down. It might sound obvious but, reliability plus innovation really are the dream ingredients for a startup. 

The collaboration and insights we’ve gained from everyone we’ve met along the way, not only from the Google Cloud team, but the amazing parents, educators, and kids who make up our community have truly made the journey so incredible. I’d love to thank them all for the support and for helping us transform education for children all over the world.

If you want to learn more about how Google Cloud can help your startup, visit our pagehere to get more information about our program, andsign up for our communications to get a look at our community activities, digital events, special offers, and more.