PRACTICAL DATA INTEGRATION FOR THE WEB
Copyright © 2017 Vertical Slice
Contents
JSON Is a Standard 3
A Brief Sample 4
Why JSON? 6
Core JSON 8
JSON Data Types 8
JSON Value Types 11
JSON Versions 14
JSON Comments 14
JSON File and MIME Type 14
JSON Style Guidelines 15
Our Example—MyConference 17
Our Technical Stack 17
Our Architectural Style—noBackEnd 17
Model JSON Data with JSON Editor Online 18
Generate Sample JSON Data with JSON Generator 20
Create and Deploy a Stub API 20
What We Covered? 24
What’s Next? 24
Node.js Setup 25
JSON Serialization/Deserialization with JSON.stringify() and JSON.parse() 26
The JSON Stringifier/Parser Object 26
JSON Serialization with Simple JavaScript Data Types 26
JSON Serialization with an Object and toJSON() 29
JSON Deserialization Using eval() 30
JSON Deserialization with an Object and JSON.parse() 31
JavaScript Objects and JSON 32
Node REPL 33
Where to Learn More About JavaScript Objects 35
Unit Testing with a Stub API 35
Unit Test Style—TDD and BDD 35
Just Enough Unit Testing with Mocha and Chai 36
Setting Up the Unit Test 36
Unirest 36
Test Data 37
Speakers Unit Test 37
Building a Small Web Application 39
Yeoman 39
Iteration 1—Generate a Web Application with Yeoman 41
Iteration 2—Make an HTTP Call with jQuery 45
Iteration 3—Consume Speaker Data from a Stub API and Use a Template 49
How to Go Deeper with JavaScript 54
What We Covered 55
What’s Next? 55
Ruby on Rails Setup 57
Ruby JSON Gems 58
JSON Serialization/Deserialization with MultiJson 58
The MultiJson Object 58
JSON Serialization/Deserialization with Simple Ruby Data Types 59
JSON Deserialization with Objects and MultiJson 61
A Word on Camel Casing and JSON 63
JSON Serialization with Objects and ActiveSupport 64
JSON Deserialization with Objects and ActiveSupport 65
Unit Testing with a Stub API 66
Just Enough Unit Testing with Minitest 66
Setting Up the Unit Test 67
Test Data 68
JSON and Minitest Testing with APIs 68
Speakers Unit Test 68
Further Reading on Ruby and Minitest 72
What Is Missing in the Unit Tests? 72
Build a Small Web API with Ruby on Rails 73
Choose a JSON Serializer 73
speakers-api-1—Create an API with Camel-Cased JSON 75
speakers-api-2—Create an API that Customizes the JSON Representation 82
Further Reading on Rails and Rails-based APIs 84
What We Covered 84
What’s Next? 84
Java and Gradle Setup 85
Gradle Overview 85
Just Enough Unit Testing with JUnit 87
Java-Based JSON Libraries 87
JSON Serialization/Deserialization with Jackson 88
Serialization/Deserialization with Simple Java Data Types 88
Serialization/Deserialization with Java Objects 91
Unit Testing with a Stub API 96
Test Data 96
JSON and JUnit Testing with APIs 96
Build a Small Web API with Spring Boot 100
Create the Model 101
Create the Controller 103
Register the Application 104
Write the Build Script 105
Deploy the API 107
Test the API with Postman 107
What We Covered 108
What’s Next? 109
JSON Schema Overview 113
What Is JSON Schema? 113
Syntactic Versus Semantic Validation 114
A Simple Example 114
JSON Schema on the Web 115
Why JSON Schema? 116
My Journey with JSON Schema 117
The Current State of the JSON Schema Standard 117
JSON Schema and XML Schema 117
Core JSON Schema—Basics and Tooling 118
JSON Schema Workflow and Tooling 118
Core Keywords 120
Basic Types 121
Numbers 125
Arrays 126
Enumerated Values 128
Objects 129
Pattern Properties 131
Regular Expressions 133
Dependent Properties 135
Internal References 136
External References 138
Choosing Validation Rules 141
How to Design and Test an API with JSON Schema 146
Our Scenario 146
Model a JSON Document 146
Generate a JSON Schema 148
Validate the JSON Document 151
Generate Sample Data 152
Deploy a Stub API with json-server 155
Final Thoughts on API Design and Testing with JSON Schema 157
Validation Using a JSON Schema Library 157
Where to Go Deeper with JSON Schema 158
What We Covered 158
What’s Next? 158
Why JSON Search? 159
JSON Search Libraries and Tools 160
Honorable Mention 160
What to Look For 160
Test Data 161
Setting Up Unit Tests 162
Comparing JSON Search Libraries and Tools 163
JSONPath 163
JSON Pointer 170
jq 173
JSON Search Library and Tool Evaluations—The Bottom Line 184
What We Covered 185
What’s Next? 185
7. JSON Transform 187
Types of JSON Transformation 187
What to Look For in a JSON Transform Library 188
Test Input Data 189
JSON-to-HTML Transformation 191
Target HTML Document 191
Mustache 192
Handlebars 198
JSON-to-HTML Transformation Evaluations—The Bottom Line 204
JSON-to-JSON Transform 204
The Issues 205
JSON-to-JSON Transform Libraries 205
Honorable Mention 205
Target JSON Output 206
JSON Patch 207
JSON-T 213
Mustache 217
Handlebars 219
JSON-to-JSON Transformation Evaluations—The Bottom Line 221
JSON-XML Transformation 222
JSON-XML Transformation Conventions 222
The Issues with JSON-XML Transformation Conventions 231
XML-JSON Transform—The Bottom Line 231
JSON-XML Transformation Unit Test 233
What We Covered 235
What’s Next? 235
Comparing Hypermedia Formats 240
Defining Key Terms 241
My Opinion on Hypermedia 241
Siren 242
JSON-LD 244
Collection+JSON 249
json:api 250
HAL 254
Conclusions on Hypermedia 259
Recommendations for Working with Hypermedia 260
Practical Issues with Hypermedia 260
Testing with HAL in the Speakers API 261
Test Data 261
HAL Unit Test 263
Server-Side HAL 267
Going Deeper with Hypermedia 268
What We Covered 268
What’s Next? 268
What About BSON? 269
MongoDB Setup 270
MongoDB Server and Tools 270
MongoDB Server 270
Importing JSON into MongoDB 271
MongoDB Command Shell 273
Basic CRUD with mongo 274
Exporting from MongoDB to a JSON Document 277
What About Schema? 280
RESTful API Testing with MongoDB 281
Test Input Data 282
Providing a RESTful Wrapper for MongoDB 282
What We Covered 285
What’s Next? 285
Kafka Use Cases 288
Kafka Concepts and Terminology 288
The Kafka Ecosystem—Related Projects 289
Kafka Environment Setup 290
Why Do I Need ZooKeeper? 290
Kafka Command-Line Interface (CLI) 291
How to Publish a JSON Message with the CLI 291
Start ZooKeeper 291
Start Kafka 292
Create a Topic 292
List Topics 293
Start a Consumer 293
Publish a JSON Message 294
Consume a JSON Message 295
Clean Up and Shut Down Kafka 295
Kafka Libraries 297
End-to-End Example—Speaker Proposals at MyConference 297
Test Data 297
Architecture Components 299
Set Up the Kafka Environment 300
Set Up Fake Email Server and Client—MailCatcher 301
Set Up Node.js Project Environment 302
Speaker Proposal Producer (Send Speaker Proposals) 302
Proposal Reviewer (Consumer/Producer) 302
Speaker Notifier (Consumer) 308
Review Notification Email Messages with MailCatcher 313
What We Covered 315
A. Installation Guides 317
B. JSON Community 335
Index 337
Preface
JavaScript Object Notation (JSON) has become the de facto standard for RESTful interfaces, but an ecosystem of little-known standards, tools, and technologies is available that architects and developers can start using today to build well-designed applications. JSON is more than just a simple replacement for XML when you make an AJAX call. It is becoming the backbone of any serious data interchange over the internet. Solid standards and best practices can be used to harness the energy and enthusiasm around JSON to build truly elegant, useful, and efficient applications.
The only thing missing is a book to pull everything together. This book aims to help developers leverage JSON so that they can build enterprise-class applications and services. My goals are to promote the use of JSON tooling and the concept of message/document design as a first-class citizen in the fast-growing API community.
My journey into JSON began in 2007 when I was leading a large web portal project, and we had to populate a drop-down list with several thousand entries. At that time, I was reading Head First AJAX by Rebecca Riordan (O’Reilly), so I had a decent architectural approach. AJAX would solve overall latency and page load issues, but what about the data? I had been using XML successfully for several years, but it seemed like overkill for the task at hand—moving data from the backend of a web application to the View. Head First AJAX mentioned a new data format called JSON, and it looked like the way to go. My team began looking into APIs that would convert our Java objects into JSON, and chose the one that had the simplest and shortest JUnit tests—the goal was to do the simplest thing that could possibly work. We put the application under rigorous load testing, and the Java-to-JSON conversion was never a performance issue. The application scaled up in production, and the users saw their drop-down list in a timely manner.
Along my journey, I considered the use of JSON with web applications, RESTful APIs, and messaging. As of 2009, I was still working with XML because XML Schema provided the semantic validation needed for meaningful data interchange. So, my position at that time was to use JSON for web user interfaces, or UIs (for speed), and XML for Web Services and Messaging (for integration). But then I heard about JSON Schema in 2010, and found that I had no further need for XML. The JSON Schema specification is still under development, but it’s sufficiently mature enough now to use for enterprise-class integration.
At this point, I was hooked on or, more accurately, obsessed with JSON. I began looking around the internet to see what else JSON could do, and I found copious APIs, online tools, search capabilities, and more. In short, anything that has been done with XML can (and should) now be done with JSON.
I then began to look for JSON in books, and was disappointed when I could find only a chapter or two on the topic in a JavaScript or RESTful Web Services book. I saw a growing JSON community along with lots of tool support and articles and blogs, but there was no single place—other than Douglas Crockford’s JSON site—that pulled everything together.
Audience, Assumptions, and Approach
This book is for architects and developers who design/implement web and mobile applications, RESTful APIs, and messaging applications. Code examples are in Java‐ Script, Node.js, Ruby on Rails, and Java. If you’re a Groovy, Go, Scala, Perl, Python, Clojure, or C# developer, you’ll need to follow along with the code examples provided.
But rest assured that most major/modern languages provide excellent JSON support. For the architect, I’ve provided guidelines, best practices, and architecture and design diagrams where appropriate. But in addition to providing visionary leadership, real architects prove their ideas with working code. While I love working with JSON and writing code, it’s entirely meaningless without use cases, and a business and technical context. For developers, this book is packed with code examples, tooling, and Unit Tests, along with a GitHub repository (see “Code Examples” on page xvii).
Chapters 5–10 only have code examples only in Node.js to keep things simple and focused. But it’s not hard to translate these examples into your platform of choice.
What Does “At Work” Mean?
When I wrote JBoss at Work with Scott Davis back in the mid-2000s, our vision was to write a book that developers could use at work on their daily jobs. In the same manner, the purpose of JSON at Work is to provide practical examples to developers based on my real-world integration experience with JSON. To that end, I’ve baked Unit Testing (wherever feasible) into every chapter. It’s simple: if there’s no test for a piece of code, then that code doesn’t exist. Period.
Expect to roll up your sleeves and look at code. Whether you’re an architect or developer, you’ll find something here to help you on your job.
What You’ll Learn
By reading and following this book’s examples, you’ll learn how to do the following:
• JSON basics and how to model JSON data
• Use JSON with Node.js, Ruby on Rails, and Java
• Structure JSON documents with JSON Schema to design and test APIs
• Search the contents of JSON documents with JSON Search tools
• Convert JSON documents to other data formats with JSON Transform tools
• Use JSON as part of an enterprise architecture
• Compare JSON-based Hypermedia formats, including HAL and json:api
• Leverage MongoDB to store and access JSON documents
• Use Apache Kafka to exchange JSON-based messages between services
• Use freely available JSON tools and utilities to simplify testing
• Invoke APIs in your favorite programming language with simple utilities and libraries
What You’ll Work With
Here’s a sample of the JSON tooling you’ll use in this book:
• JSON editors/modelers
• Unit-Testing tools (e.g., Mocha/Chai, Minitest, JUnit)
• JSON Validators
• A JSON Schema Generator
• JSON Search tools
• JSON Transform (templating) tools
Who This Book Is Not For
This book is not for you if your only interest in JSON is to make AJAX calls from JavaScript. Although I cover this topic, it’s just the tip of the iceberg. Plenty of Java‐ Script books have the chapter you’re looking for.
Developers looking for a deep reference on REST, Ruby on Rails, Java, JavaScript, etc. won’t find it here. This book relies on these technologies, but focuses on how to use JSON with these languages and technologies.
Organization
This book consists of the following parts:
• Part I, JSON Overview and Platforms
• Part II, The JSON Ecosystem
• Part III, JSON in the Enterprise
• Appendices
Part I, JSON Overview and Platforms
• Chapter 1, JSON Overview, starts with an overview of the JSON data format, describes best practices in JSON usage, and introduces the tools used throughout the book.
• Chapter 2, JSON in JavaScript, shows how to use JSON with JavaScript, Node.js, and Mocha/Chai Unit Tests.
• Chapter 3, JSON in Ruby on Rails, describes how to convert between Ruby objects and JSON, and integrate with Rails.
• Chapter 4, JSON in Java, tells you how to use JSON with Java and Sprint Boot.
Part II, The JSON Ecosystem
• Chapter 5, JSON Schema, helps you structure JSON documents with JSON Schema. Along the way, you’ll generate a JSON Schema and design an API with it.
• Chapter 6, JSON Search, shows how to search JSON documents with jq and JSONPath.
• Chapter 7, JSON Transform, provides the tools you’ll need transform a poorly designed JSON document to a better designed/more useful JSON document. Plus, it shows how to convert between JSON and other formats such as XML and HTML.
Part III, JSON in the Enterprise
• Chapter 8, JSON and Hypermedia, looks at how to use JSON with several wellknown Hypermedia formats (e.g., HAL and jsonapi).
• Chapter 9, JSON and MongoDB, shows how to leverage MongoDB to store and access JSON documents.
• Chapter 10, JSON Messaging with Kafka, describes how to use Apache Kafka to exchange JSON-based messages between services.
JSON Overview
The JavaScript Object Notation (JSON) data format enables applications to communicate over a network, typically through RESTful APIs. JSON is technology-agnostic, nonproprietary, and portable. All modern languages (e.g., Java, JavaScript, Ruby, C#, PHP, Python, and Groovy) and platforms provide excellent support for producing (serializing) and consuming (deserializing) JSON data. JSON is simple: it consists of
developer-friendly constructs such as Objects, Arrays, and name/value pairs. JSON is not limited to Representational State Transfer (REST); it also works with the following:
• Node.js (which stores project metadata in package.json)
• NoSQL databases such as MongoDB (see Chapter 9)
• Messaging platforms such as Kafka (see Chapter 10)
JSON Is a Standard
In the early days, REST’s detractors derided RESTful Web Services as being nonstandard, but (just like HTTP) JSON is in fact a standard. Both the Internet Engineering Task Force (IETF) and Ecma International (formerly the European Computer Manufacturers Association, or ECMA) have recognized JSON as a standard.
Douglas Crockford originally created JSON in 2001, and initially standardized it in 2006 under RFC 4627 through the IETF; see the JSON specification. In the fall of 2013, Ecma International also standardized JSON under ECMA 404; see their JSON specification. With Ecma recognition (per Douglas Crockford; see his Google+ page), JSON is now considered a formal international data processing standard.
