Remote Service API Design

With the longer weekend (Friday Nov 11 is a public holiday in Belgium, to remember the end of WW I) took some time to watch some sessions from the Devoxx conference that has just finished. The talks are already available on Youtube! Great content: Microservices, Reactive, Netty, Kafka messaging, Docker, ... With this conference almost in my backyard, will have to free up some time next year to attend again myself.

One talk immediately drew my attention: "Effective Service API Design" by Elliott Rusty Harold. I know Elliott an XML guru. Remember being at the speaker's table with him at XML DevCon in London back in 2001 where I presented "Understanding SOAP".

Some thinks I picked up:

• "Contract first" >> "Documentation first"
• Start small (MVP Minimum Viable Product, YAGNE You Ain't Gonna Need It)
• Leverage the URL, not everything 
• Prefer idempotency
• Use standards: UTF-8, standard data/date formats, standard decimals with 2 digits 
• Avoid required data elements
• Deprecation policy, how long to keep the old API, 2 year notice is good
• Plan for versioning: easier without schema's but with optional fields
• No assumption about use of client code/library, developers will often not use those 
• If you build client code/library, develop it by hand and do not generate it 
• Authentication and authorization remain a challenge
• Performance!

Although a good talk, would have hoped for some more specific guidelines.

 
•  1:00:41

  Effective Service API Design by Elliotte Rusty Harold

At the start of the presentation, Elliott clearly makes the difference between the local "library API" (e.g. the JDBC API with the java.sql package) and "Remote API" or "Service API". Great definition by the way: a network service (almost always a server) by which programs communicate with a bundle of funcitonality provided by code owned by somone else, running on their computer (not yours).

Thanks to the Devoxx team and Stephan Janssen to publish this great content for free!

Reactive programming, the old way

Reactive programming is the new hit: applications work with a non-blocking, asynchronous programming model. Use of multi-threading is none or very limited. The reactive program responds to messages or events that come in via callbacks. These messages must be handled as quickly as possible, never blocking themselves while being processed.

Made me think of a fun project I was involved with in 1999 and 2000: "GSL", the Generic Service Layer" at Rabobank. We developed an integration layer (by Gartner defined as a "Super API") based on the message passing product Netweave. And this product was completely based on callbacks and a reactive programming model.

Pre-dating XML and with maximum message buffer size of 32K on CICS, we defined our own message format. The messages had a tree structure in a proprietary, textual format. The GSL API allowed the creation, sending, receiving and parsing messages.

All the main platforms were supported: IBM CICS, Tandem (Guardian), AIX Windows NT and AIX. Communication was supported in all directions. So yes, a COBOL CICS program could invoke an ActiveX component on Windows. Code was written in C (and bit of C++). Most often the IBM mainframe and Tandem would be the actual servers.

An old code snippet from those days. This piece of C code writes a buffer back to a client application.

void loclSend(client_t * pClient, size_t szBuf, char * pBuf)
{
  NWDS_ERRNO      retcode  ;
  NWDS_CALL_BACK  complete ;

  /* write data to client process */
  complete.procedure = loclSendComplete ;
  complete.context   = (NWDS_CONTEXT) pClient ;
  retcode = nwds_ipc_write(pClient->hLocl
                          ,(NWDS_SIZE) szBuf
                          ,(void *)    pBuf
                          ,NULL
                          ,&complete) ;
  if ((retcode == NWDS_PENDING) ||
      (retcode == NWDS_SUCCESSFUL))
    lcTraceHdr('I', 'S', pBuf, szBuf) ;

  if (retcode != NWDS_PENDING)
    loclSleepCallback(&complete,retcode) ;
}

The write of the buffer is executed with ndws_ipc_write. This write could return immediately, whereby NDWS_SUCCESSFUL would be returned. But most often, the return code would be NWDS_PENDING. Then the callback function loclSendComplete would be registered and executed when the write was finished. The complete.procedure element contains a pointer to the loclSendComplete() function.

IT Compass

During a meeting, the terms "north" and "south" popped up again while discussing network zoning and DMZ. North is the "outside", typically the Internet. South are the internal, trusted systems.

North-South in networking (from PaloAlto networking)

How would IT people in South-Africa think about this, where the north is "hot" and very south Antarctica cold? Maybe also good fit: freezing in the data center, hot and dangerous on the Internet.

Unikernel: microservices without an OS

As I continue to spend lots of time on the road and in traffic jams, podcasts are an efficient way to keep up-to-date in the fast moving IT world. Just finished listening SE-Radio Episode 271: Idit Levine on Unikernels. Very interesting.

Had never heard of Unikernel. Actually this is hardly a real OS: no multi-tasking, no security, no memory management. Memory directly mapped to the underlying hardware and some device drivers.

Unikernels become most relevant when used on top of a virtualization layer. A single application is combined with the Unikernel to become a super light-weight runtime unit. A fine alternative for microservices running in a container.

Unikernels are also a nice fit with server-less architectures: the Unikernel App is super-fast to start. So Unikernel Apps as a more efficient Function-As-A-Service approach.

When mapping this to my own world of integration, the Unikernel App could be a nice mechanism to handle all the async events going on in an integration environment. An incoming message or API call starts the Unikernel App which handles the message: transformation, routing, logging, forwarding...

Incredible at which pace the IT world keeps moving.

Event-driven Microservices

A few weekends ago, I decided that it was time to catch up on Microservices. Hadn't been following the topic for a longer while. Starting point is often Safari: but instead of reading, I ended up watching the video tutorial "Event-Driven Microservices". By the way, the first part is freely accessible without a Safari subscription.

The video course is presented by Chris Richardson. I know Chris as a speaker at earlier Devoxx conferences. Very interesting is his approach to address the topic based on patterns, documented at microservices.io.

Microservices patterns (from Microservices.io, by Chris Richardson)

What was new to me and most interesting was the part about Event-Sourcing: instead of storing the state of each object as a database row, the sequence of changes is recorded and maintained. This an approach to tackle the issue of distributed transactions that do not fit with a Microservices architecture. This is also the topic Richardson personally focuses on as his latest startup is all about Event Sourcing.

List of events represents state (Source eventuate.io)

Handling request by rebuilding state from list of events (Source: eventuate.io)

Of course one needs to remain critical about Microservices:

• We've been successfully building 3-tier web and mobile apps, why do it all different now?
• How heterogeneous will all these microservices be?  
• Not every organisation is like LinkedIn or Netflix
• Transactions that are eventually consistent are not trivial
• What will be the next thing after Microservices?

After 4h 47minutes, I understood that Microservices is as well a domain that is in full flux.