The P-Tumati Distribution

Category: web sockets

  • My First Experience with Server-Side Swift (using Perfect from perfect.org) – Part 1

     

    Motivations

    Most of the work I take on involves languages like C++, Swift, or Objective-C and is not typically web based.  While I’m normally nestled comfortably in the embrace of C++ template meta-programming or the warm walls of an operating system kernel, even I can sense the uncivilized barbarians at the gate (Javascript) and the increasing demand that software work more closely with web based APIs and various web stacks (Django, Rails, node.js.)

    Recently, I had the interesting encounter where I was given the chance to explore the use of server-side Swift.  The client was attempting to ferry computational requests from a number of web and iOS-based clients to a very large, computational backend written in C++ (complete with its own ORM)  that was distributed across a number of nodes.  Since one of the core libraries used happened to be written in Swift, there was some hope we could simply use code on the server side without much effort.  Rather than compile a separate application and marshal requests to the Swift library through yet another framework, the decision was made to just service requests outright from a web service authored in Swift.

    Instead of authoring web services from scratch, I looked for a ready-made solution to simply embrace and extend.  There were a few different frameworks to choose from, but I eventually settled on Perfect (from http://www.perfect.org).  The decision to use a server-side swift framework, given other mature technologies, was not without internal friction.  Criticism was leveled that I was using untested “hipster” technology.  In fact, some of my colleagues insisted I go so far as to don a beret, head to the local hipster coffee shop, and offer poorly authored poetry to the public about being oppressed by self-authored technical debt!  Despite critics’ concerns, I want to highlight that I had some success with Perfect!  (Also, not enough technical debt was accumulated to warrant lobbing harangues-inappropriately-called-poetry at the public.)  Of course, the effort was not completely glitch free — and that is part of what I will document here.

    Why Perfect?

    Why Perfect?  My choice was based on a mix of inputs:  GitHub commits, looking into public issue trackers, various blogs, popularity, and out of the box features.  I watched a video on YouTube from the CEO of the organization responsible for Perfect and thought his developers’ take on things aligned with my own views.  Moreover,  in my particular environment, I needed connectors for Redis and Postgres.  The support for web sockets and the basic documentation gave me enough confidence in the product to give it a shot.  If Perfect failed me, I figured, it would fail me quickly and in an obvious enough fashion that I could bail without disastrous results.

    For our internal use case, some “jankiness” in the tech stack was tolerable provided we got results often enough between failures.  I was willing to fix bugs, provided they weren’t too deep rooted.  Basically, Perfect really needed to be “good enough” — and for our use case, it was.

    The Use Case

    Our use case:  Accept work requests from web, desktop, or mobile clients, perform some transformations and filtering on those requests, service those requests, or (in the case of computational expensive requests) dispatch those requests to a work queue (Redis).  Binaries written in C++ would then consume elements from Redis and place their results back in a database (Postgres) or, in some cases, Redis.  As the computational backend progressed on its work requests, updates were provided via WebSockets (also via Perfect) to web based clients.

    Well, did you succeed?

    I did!  In fact, the general architecture of Perfect facilitated our needs quite well.  I would say the initial prototyping and deployment of a basic service went very smoothly.  The transition from Swift 3 to Swift 4 even went smoothly — the developers responsible for Perfect handled this transition in a very timely fashion. There was no extended period in which I had clients on one version of Swift with the server on another version of Swift.

    Routes, URL Variables, Responses

    In Perfect, it is very easy to create a server, add routes, and service requests.  No particularly advanced knowledge of Swift is really required to accomplish this.  Given my experience with other frameworks, I did not encounter too much difficulty just getting up and running.  The basic process is to set up a route, pass a function parameter to handle the route, and reply using response objects.  Most of the basic web primitives for handling URL variables, posted parameters, and managing responses are provided.

    Implementing an API is simple.  Return the right response type, format your response with the proper encoding, and write the response.  I hit no major issues or bugs in performing these basic tasks.

    Encouraging!

    Using Swift on the Server Side in Linux

    Swift on the server-side does have some gotchas — this isn’t Perfect’s fault, but it affected me in that, if I developed code in Xcode and then recompiled and deployed on Linux, some small snippets of code would not compile.

    In addition, the Foundation libraries, while functionally complete, have some corner cases where things can get awkward.  One (cosmetic) thing I found somewhat unusual was the use of NSRegularExpression.  OS X and iOS developers will normally see “NS” namespace objects, but this felt slightly awkward on Linux.  Moreover, NSRegularExpression (and affiliated functions) is less pleasant to use than regex facilities in other languages.  (Obviously, this is an opinion; but I suspect many readers who’ve used other frameworks would agree with me.)

    During the transition between Swift 3 and 4, I hit some snags with certain function signatures changing.  In some cases, my code was littered with preprocessor defines like #if os(OS X) to tweak behavior on different platforms.  I didn’t love this, but it wasn’t too difficult to work around and was only a small drawback.

    Another issue I encountered was doing semi- low-level tasks — I ended up needing third party libraries to do things like atomically incrementing integers.   I found this somewhat distasteful, but not unreasonable given the nature of how Swift has been evolving and changing over the past few years.  Some primitives available on OS X were not available on Linux and suggestions on Stack Overflow seemed clunky.  I ended up using a third-party library that made effective use of features provided by Clang.  Performing some systems-software tasks in Swift still feels awkward compared to using C or C++.  This could be my own experience, but it certainly felt awkward and I’d caution others who have these same scenarios.

    Interfacing to Postgres

    I did not use the ORM (stORM) provided with Perfect.  I did, however, issue SQL requests via their Postgres package.  I think, in many languages and frameworks, authoring SQL requests is a bit of a pain.  It wasn’t any different in Swift while using Perfect.  I’m not sure there’s a way around painfully constructing strings representing complex queries, but there’s no immediately perceptible or dramatically noticeable edge in Swift for performing this task.

    One issue that did cause me to raise an eyebrow was a bug filed in the Perfect issue tracker regarding memory leaks on tables returned from the Perfect driver built on top of libpq.  My use case did not stress the Postgres driver enough for me to experience anything catastrophic with regard to using Postgres.  At the time of writing for this blog post, this issue has not been closed.  This could be of concern for long running services issuing a very large number of requests.  Hopefully the developers will address this.

    What’s Coming in Part 2

    In part 2, I’ll detail a few issues I hit in Perfect-Mustache, issues I encountered in the use of Perfect-Redis, and other architectural gotchas that I encountered.  The issues I touch upon in Part 2 have more substance.  Stay tuned!

     

  • C++, WebSockets, and Mt.Gox

    I’ve been reading quite a bit about websockets lately. To test my understanding of the concepts involved, I was trying to connect an application to Mt.Gox in order to collect bitcoin-related pricing information. I was having some difficulty getting all of the parameters aligned properly with all of the various websocket libraries, so I went through the process of reading other people’s code on github, sifting through stackoverflow-style-forums, and experimenting until things worked correctly. I thought I’d document some of the minor hurdles here, just to save some people out there in the wild some effort.

    I tried a few approaches:

    • Python using autobahn
    • C++ using libwebsockets (a C library)

    The Python implementation worked perfectly.  The only quirk was that I had to trick the application into inserting an “Origin:” field in the websocket handshake in order to get the application to connect.  (This had something to do with something called the Web Origin Concept.  It’s a safety mechanism for web browsers.)  After tweaking the code to force an ‘Origin’ using the autobahn library, the handshake with the server ended up looking something like this:

    GET /mtgox HTTP/1.1
    User-Agent: disease
    Origin: websocket.mtgox.com:80
    Host: websocket.mtgox.com:80
    Upgrade: WebSocket
    Connection: Upgrade
    Pragma: no-cache
    Cache-Control: no-cache
    Sec-WebSocket-Key: JI2o83NARHJDps1XCN7eCQ==
    Sec-WebSocket-Version: 13

    After a handshake of this style, data starts streaming over the connection with no issue. With a working Python version, I had enough knowledge to craft an appropriate handshake in the C++ version of my bitcoin application.

    With libwebsockets, it took some tinkering with the source code to get things to work right with Mt.Gox. The extensions field in the ‘lws_context_creation_info’ should not be filled in, and in the connection phase in libwebsocket_client_connect(), the specified protocol should be NULL. The origin and host strings fed into the libwebsocket_client_connect() function should mirror what you see above in the successful python handshake.

    I didn’t bother to post code here, but the general idea to get things to start streaming correctly is to have a handshake that looks something like what I’ve copied above for the python implementation I used.

    I hope this helps the random Googler out there searching for information.

  • jQuery, Websockets, and Bitcoin Time and Sales

    If you haven’t seen Clark Moody’s Mt.Gox bitcoin market data website yet, you should.  It’s a really beautiful presentation of realtime Mt.Gox BitCoin orderbook depth, the trade tape, and charts at various time frames.

    I was so impressed by Clark’s presentation of market information that I wanted to dig into how modern web sites present information from real-time data sources.  I don’t typically work on web applications, so I had to figure out what was involved.  There were three pieces of key knowledge:  1) Basic javascript, 2) knowing how to use a websocket API, and 3) using CSS and jquery to figure out how to manipulate documents in a web browser.  (I had very minimal knowledge of all three.  Fortunately, Javascript is easy to get a handle on for someone with previous programming experience.)

    I first went to look at the Mt.Gox API web page, where I found some simple explanations on how to connect to their streaming market data.  After using a dozen console.log() statements to see what I was looking at, I finally tweaked my Javascript code to manipulate table rows in a tbody styled by CSS.  I ended up with this time and sales viewer.  (Please note that when loading the viewer, it will take a few seconds to establish a connection to the exchange.  Once the connection occurs, you should be able to see current trades happening in realtime.  Note that I’ve only tried the viewer in Chrome and Firefox.  I have no idea what browsers it works and doesn’t work in, as the project was just an experiment.)

    The mini-project was a great deal of fun — websockets are very interesting.  I was impressed by how much could be accomplished with so little work.