GCP – Spanner integration testing with the emulator
Spanner is a highly scalable, reliable, and globally distributed database from Google Cloud ideally suited for business-critical applications that demand high performance and continuous operation.
As a developer, you need thorough testing to ensure the seamless integration of Spanner into your applications. Integration testing focuses on verifying that different components of a system work together after making changes to those components. For Spanner, integration testing ensures that your application’s data operations such as transactions and error handling work correctly with the database.
This post demonstrates how to set up integration testing for Spanner using GitHub Actions and the Spanner emulator. The emulator mimics the behavior of Spanner outside of Google Cloud, which is helpful for rapid development of applications backed by a Spanner database.
The example application we’ll test is a Golang backend service that manages player profiles for a fictitious game. However, these principles can be used for other applications and services in other languages and industries.
The “integration” we are testing here is between the profile service and Spanner for a fast feedback loop to ensure code changes to the service will work correctly. This is not full end-to-end testing between all services in our stack. Doing testing at that level should use an actual staging environment with Spanner prior to deploying to production.
You can find out more in this post which has a good overview of the types of tests that should be performed to qualify a software release.
We’ll look at these components for our integration tests:
GitHub Actions automates execution of tests and is built into the platform where our code is located. Other CI/CD platforms will work similarly.Spanner emulator as a lightweight and offline emulation of a Spanner databaseProfile Service is our application that depends on Spanner.
More details on these components are provided below, but the architecture will look something like this:
Integration testing profile-service using Spanner emulator
GitHub Actions: Automating your workflow
Since our service code is stored in a GitHub repository, GitHub Actions are the perfect option for our automated integration tests.
GitHub Actions is part of a continuous integration and continuous delivery (CI/CD) platform that automates your software development workflow. It integrates seamlessly with GitHub repositories, allowing you to define and execute automated tasks triggered by code changes or scheduled events.
The Spanner emulator: A local testing environment
The Spanner emulator is a lightweight tool that can run completely offline. This enables developers to test their applications against Spanner without incurring any cloud costs or relying on an actual Spanner instance. This facilitates rapid development cycles and early detection of integration issues.
There are some differences and limitations to the Spanner emulator compared to an actual Spanner database that you should be familiar with.
Setting up integration testing for the profile service
The code for the sample gaming application can be found on Github. We will look first at the integration test for the profile service, and then the workflow that enables automated integration testing using Github Actions.
The profile service integration test can be found in the profile-service’s main_test.go file. This file contains the following sections:
Starting the Spanner emulator.Set up the Spanner instance and database with the schema and any test data needed.Set up the Profile serviceThe tests themselves.Cleaning up after the tests complete
Starting the Spanner emulator
Because the Spanner emulator is deployed as a container, we use the testcontainers-go library. This makes it extremely easy to codify starting the emulator:
<ListValue: [StructValue([(‘code’, ‘var TESTNETWORK = “game-sample-test”rnrnrntype Emulator struct {rn testcontainers.Containerrn Endpoint stringrn Project stringrn Instance stringrn Database stringrn}rn*snip*rnfunc setupSpannerEmulator(ctx context.Context) (*Emulator, error) {rn req := testcontainers.ContainerRequest{rn Image: “gcr.io/cloud-spanner-emulator/emulator:1.5.0”,rn ExposedPorts: []string{“9010/tcp”},rn Networks: []string{rn TESTNETWORK,rn },rn NetworkAliases: map[string][]string{rn TESTNETWORK: []string{rn “emulator”,rn },rn },rn Name: “emulator”,rn WaitingFor: wait.ForLog(“gRPC server listening at”),rn }rn spannerEmulator, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{rn ContainerRequest: req,rn Started: true,rn })rn if err != nil {rn return nil, errrn }rnrnrn // Retrieve the container IPrn ip, err := spannerEmulator.Host(ctx)rn if err != nil {rn return nil, errrn }rnrnrn // Retrieve the container portrn port, err := spannerEmulator.MappedPort(ctx, “9010”)rn if err != nil {rn return nil, errrn }’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc4bfda7c0>)])]>
This sets up the emulator container with a mapped port of 9010 that we can communicate with. The networking uses a Docker network, so that any container or process with access to that network can communicate with the ’emulator’ container.
The testcontainers-go library makes it easy to wait until the container is ready before proceeding to the next steps.
When it is ready, we capture the host information and expose that as an operating system environment variable and define a golang struct. Both of these will be used later for setting up the instance and database.
<ListValue: [StructValue([(‘code’, ‘// OS environment needed for setting up instance and databasern os.Setenv(“SPANNER_EMULATOR_HOST”, fmt.Sprintf(“%s:%d”, ip, port.Int()))rnrnrn var ec = Emulator{rn Container: spannerEmulator,rn Endpoint: “emulator:9010”,rn Project: “test-project”,rn Instance: “test-instance”,rn Database: “test-database”,rn }’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc4bfda9a0>)])]>
When all that is ready, we can create the Spanner instance and database:
<ListValue: [StructValue([(‘code’, ‘// Create instancern err = setupInstance(ctx, ec)rn if err != nil {rn return nil, errrn }rnrnrn // Define the database and schemarn err = setupDatabase(ctx, ec)rn if err != nil {rn return nil, errrn }rnrnrn return &ec, nilrn}’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc4bfda9d0>)])]>
Setup the Spanner instance and database
With the emulator running, we need to set up a test instance and database. First, let’s set up the instance:
<ListValue: [StructValue([(‘code’, ‘func setupInstance(ctx context.Context, ec Emulator) error {rn instanceAdmin, err := instance.NewInstanceAdminClient(ctx)rn if err != nil {rn log.Fatal(err)rn }rnrnrn defer instanceAdmin.Close()rnrnrn op, err := instanceAdmin.CreateInstance(ctx, &instancepb.CreateInstanceRequest{rn Parent: fmt.Sprintf(“projects/%s”, ec.Project),rn InstanceId: ec.Instance,rn Instance: &instancepb.Instance{rn Config: fmt.Sprintf(“projects/%s/instanceConfigs/%s”, ec.Project, “emulator-config”),rn DisplayName: ec.Instance,rn NodeCount: 1,rn },rn })rn if err != nil {rn return fmt.Errorf(“could not create instance %s: %v”, fmt.Sprintf(“projects/%s/instances/%s”, ec.Project, ec.Instance), err)rn }’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc4bfda8b0>)])]>
This leverages the Spanner instance golang library to create the instance. This only works because we set the SPANNER_EMULATOR_HOST environment variable earlier. Otherwise, the Spanner library would be looking for an actual Spanner instance running on your Google Cloud project.
Now, we wait until the instance is ready before proceeding in our test.
<ListValue: [StructValue([(‘code’, ‘// Wait for the instance creation to finish.rn i, err := op.Wait(ctx)rn if err != nil {rn return fmt.Errorf(“waiting for instance creation to finish failed: %v”, err)rn }rnrnrn // The instance may not be ready to serve yet.rn if i.State != instancepb.Instance_READY {rn fmt.Printf(“instance state is not READY yet. Got state %v\n”, i.State)rn }rn fmt.Printf(“Created emulator instance [%s]\n”, ec.Instance)rnrnrn return nilrn}’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc65ec8580>)])]>
For the database setup, we need a schema file. Where this schema file comes from is up to your processes. In this case, I make a copy of the master schema file during the ‘make profile-integration‘ instructions in the Makefile. This allows me to get the most recent schema that is relevant to player profiles.
To set up the database, we leverage Spanner’s database golang library.
<ListValue: [StructValue([(‘code’, ‘//go:embed test_data/schema.sqlrnvar SCHEMAFILE embed.FSrnrnrnfunc setupDatabase(ctx context.Context, ec Emulator) error {rn // get schema statements from filern schema, _ := SCHEMAFILE.ReadFile(“test_data/schema.sql”)rnrnrn // TODO: remove this when the Spanner Emulator supports ‘DEFAULT’ syntaxrn // This is still a problem when using SQL to insert data. see: https://github.com/GoogleCloudPlatform/cloud-spanner-emulator/issues/101rn schemaStringFix := strings.Replace(string(schema), “account_balance NUMERIC NOT NULL DEFAULT (0.00),”, “account_balance NUMERIC,”, 1)rnrnrn schemaStatements := strings.Split(schemaStringFix, “;”)rnrnrn adminClient, err := database.NewDatabaseAdminClient(ctx)rn if err != nil {rn return errrn }rn defer adminClient.Close()rnrnrn op, err := adminClient.CreateDatabase(ctx, &databasepb.CreateDatabaseRequest{rn Parent: fmt.Sprintf(“projects/%s/instances/%s”, ec.Project, ec.Instance),rn CreateStatement: “CREATE DATABASE `” + ec.Database + “`”,rn ExtraStatements: schemaStatements,rn })rn if err != nil {rn fmt.Printf(“Error: [%s]”, err)rn return errrn }rn if _, err := op.Wait(ctx); err != nil {rn fmt.Printf(“Error: [%s]”, err)rn return errrn }rnrnrn fmt.Printf(“Created emulator database [%s]\n”, ec.Database)rn return nilrnrnrn}’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc657f31f0>)])]>
In this function, we can handle modifications to the schema to be understood by the emulator. We have to convert the schema file into an array of statements without the trailing semicolons.
After the database setup is complete we can start the profile service.
Start the profile service
Here, we are starting the profile service as another container (using testcontainers-go) that can communicate with the emulator.
<ListValue: [StructValue([(‘code’, ‘type Service struct {rn testcontainers.Containerrn Endpoint stringrn}rn*snip*rnfunc setupService(ctx context.Context, ec *Emulator) (*Service, error) {rn var service = “profile-service”rn req := testcontainers.ContainerRequest{rn Image: fmt.Sprintf(“%s:latest”, service),rn Name: service,rn ExposedPorts: []string{“80:80/tcp”}, // Bind to 80 on localhost to avoid not knowing about the container portrn Networks: []string{TESTNETWORK},rn NetworkAliases: map[string][]string{rn TESTNETWORK: []string{rn service,rn },rn },rn Env: map[string]string{rn “SPANNER_PROJECT_ID”: ec.Project,rn “SPANNER_INSTANCE_ID”: ec.Instance,rn “SPANNER_DATABASE_ID”: ec.Database,rn “SERVICE_HOST”: “0.0.0.0”,rn “SERVICE_PORT”: “80”,rn “SPANNER_EMULATOR_HOST”: ec.Endpoint,rn },rn WaitingFor: wait.ForLog(“Listening and serving HTTP on 0.0.0.0:80″),rn }rn serviceContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{rn ContainerRequest: req,rn Started: true,rn })rn if err != nil {rn return nil, errrn }’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc612523a0>)])]>
When the service is ready, we capture the endpoint information and expose that as a struct for use in our tests:
<ListValue: [StructValue([(‘code’, ‘// Retrieve the container endpointrn endpoint, err := serviceContainer.Endpoint(ctx, “”)rn if err != nil {rn return nil, errrn }rnrnrn return &Service{rn Container: serviceContainer,rn Endpoint: endpoint,rn }, nilrn}’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc61252d60>)])]>
With both the emulator and service running, we can run our tests.
Running the tests
Our integration tests use the testify assert library and hit the endpoints for our profile service. Our service tests the following behavior:
<ListValue: [StructValue([(‘code’, ‘func TestAddPlayers(t *testing.T) {rn *snip*rn}rnrnrnfunc TestGetPlayers(t *testing.T) {rn *snip*rn}rnrnrnfunc TestPlayerLogin(t *testing.T) {rn *snip*rn}rnrnrnfunc TestPlayerLogout(t *testing.T) {rn *snip*rn}’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc61252790>)])]>
Cleaning up
Once the tests are run, it’s time to clean up the containers that we created. To do this, we run a teardown function:
<ListValue: [StructValue([(‘code’, ‘func teardown(ctx context.Context, emulator *Emulator, service *Service) {rn emulator.Terminate(ctx)rn service.Terminate(ctx)rn}’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc61252700>)])]>
Once again, testcontainers-go makes it easy to clean up!
The Github Action workflow
Setting up Github Actions is as simple as adding workflow files in the .github/workflows directory of the repository.
The behavior of the Action depends on the instructions of each file. Does the action trigger on push, or for a pull request? Do changes to all files trigger the action, or only a subset? What dependencies need to be in place to run the Action?
Here is the YAML Action defined for the profile service:
<ListValue: [StructValue([(‘code’, “name: Backend Profile Service code testsrnon:rn pull_request:rn paths:rn – ‘backend_services/profile/**’rnjobs:rn test:rn name: Profile Service Testsrn runs-on: ubuntu-latestrn steps:rn – uses: actions/checkout@v3rn – uses: actions/setup-go@v3rn with:rn go-version: ‘^1.19.2’rn – name: Check go versionrn run: go versionrn – name: Lint filesrn uses: golangci/golangci-lint-action@v3rn with:rn working-directory: ./backend_services/profilern args: –timeout 120s –verbosern – name: Run unit testsrn run: |rn make profile-testrn – name: Run integration testsrn run: |rn make profile-test-integration”), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc61252400>)])]>
This simple yaml file defines the tasks to run only on a pull request that contains changes to the backend_services/profile directory. The action installs go dependencies and runs some lint checks before going on to the unit and integration tests.
The make commands for unit tests and integration tests are defined in the repository’s Makefile.
<ListValue: [StructValue([(‘code’, ‘.PHONY: profile-testrnprofile-test:rn echo “Running unit tests for profile service”rn cd backend_services/profile && go test -short ./…rnrnrn.PHONY: profile-test-integrationrnprofile-test-integration:rn echo “Running integration tests for profile service”rn cd backend_services/profile \rn && docker build . -t profile-service \rn && mkdir -p test_data \rn && grep -v ‘^–*’ ../../schema/players.sql >test_data/schema.sql \rn && go test –tags=integration ./…’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3dfc6767fac0>)])]>
Notice the integration test setting up the test_data/schema.sql file.
With this in place, when a pull request is opened with changes to the profile service the integration test looks roughly like this:
Conclusion
By leveraging the Spanner emulator and GitHub Actions, you can establish a robust integration testing environment for your Spanner applications. This approach enables you to detect and resolve integration issues early in the development process, ensuring the smooth integration of Spanner into your applications.
To further explore the capabilities of Spanner, take advantage of a free trial instance. This will allow you to experiment with Spanner and gain hands-on experience with its features and functionality at no cost for 90 days.
Read More for the details.
