I went to the Craft Conf with exact goals. I have written in my previous article Why Am I Attending The Craft Conf 2022 Software Development Conference? about them. I advise reading it before this writing as I will revisit my goals one by one and reflect on how much I was able to achieve from them.

Backstory

I remember I was a little nervous before the day of the conference. As I explained in my previous article, I am an introvert. The idea of being together with hundreds of software developers freaked me out a little bit.

Luckily I was not alone, as 4 of my friends came with me to the Craft Conf. I cannot exaggerate enough how much it meant for me. The idea of spending two days with my developer friends, talking about software development, and of course, about other friendish stuff made me happy and helped to reduce my nervousness.

I was not sure what to wear during the conference, but I was not alone with that problem as two of my friends also asked me about that. In the end, we decided to dress as we would go to work. For me, it meant jeans, a Hashnode T-shirt, and a shirt too which I took off in the end as it was pretty hot.

In today’s fast-paced world, where new technologies emerge yearly, it is very important to train yourself if you do not want to fall behind. By now, many companies have realized how important it is to support their employees by sending them to training or giving them conference tickets. It is important for the company not just because the new learning will benefit them too, but also because this can be a factor that motivates a developer to stay with the company. At least for me, it is an important factor.

Unfortunately, my previous company hardly ever supported me in improving myself, which was also why I did not attend conferences before. Finally, after 15 years, I moved to another company in December.

My new company is different as it provides a budget and 4 extra holidays for its employees to train themselves. I have already spent one day visiting the Future of Testing Framework conference, about which you can read my report here. So I could visit the Craft Conf using two days from my holiday budget. I am not exactly sure what will I do with my fourth day. Still, if I have to answer now, then I would say I will visit the Crunch Conf, which is also organized by CraftHub, but instead of software developers, it is targeting Data Engineers and Data Scientists.

What about the imposter syndrome? Luckily I left it behind around half a year ago. So I was excited to visit the conference, as instead of thinking, “I have no place amongst these much better developers,” I felt I would be in the right place with fellow developers who would be there to learn and have fun just like me.

That is enough about the backstory. Let’s start the conference.

Registration

We arrived quite early at the conference as luckily my friend Szabi brought me to the location by car. Our first journey led us to the registration booth. It was quite a surprise to see the registration is on the dodgem field. Look:

There were many desks where you were able to register. Almost nobody was there when we registered, but based on the number of staff there, I am pretty sure the fellow developers could register quickly too.

Catering

After the registration, we entered the building and headed directly to the breakfast area as both of us were hungry, especially me.

A few (at least compared to the total number) of the developers were already having breakfast, as you can see in the picture. There were sandwiches and other food too and even dessert. I am a meat lover, so I sadly realized all the sandwiches were vegetarian, but this sadness faded as soon as I tasted the sandwiches. They not just looked good, but they were delicious too.

This was the same place where we had lunch. Luckily lunch was not self-serving, as I think it would have been chaos if hundreds of hungry developers would have appeared in an all-you-can-eat area. So the food was served by the staff thanks to which you did not have to wait too much. There were chicken, pork, beef, fish, and vegan food so that everyone could find something for him/herself.

There were many stands where you could get drinks and coffee too. Luckily there were many glasses prefilled with drinks, so you could simply grab one when thirsty.

All in all, the food was delicious and as it was all-you-can-eat I am pretty sure no developer remained hungry.

Now that our stomach is full, let’s look around at the location.

The Location

Craft Conf took place at the Hungarian Railway Museum. I have to admit I did not look around thoroughly, but during a lunch break I went out to see the locomotives.

I have spent more than 10 years of my life traveling by train twice a week to visit my parents, then go back to school. To be honest, I did not like traveling, but I still love locomotives. I remember there was a beautiful black locomotive at the Mezőhegyes railway station. I loved that locomotive. We always jumped on it with my brother Tamás and played in the cockpit.

After checking the trains it is time for a cigarette.

The Smoking Area

I have to admit, all my friends and also myself are smokers. Thanks to this our meeting point was the smoking area. After each talk, we gathered together in front of the main entrance where the smoking area was located. During smoking, we discussed the previously heard talk and also used the time to have many friendly talks. For me, that was the best part of the conference. Of course, we heard amazing talks and had a lot of fun, but spending two days with my friends was invaluable.

Talks and Speakers

More than 70 speakers were invited to the Craft Conf 2022. Most of them were there personally and the others had an online talk. Arranging the speakers’ traveling, accommodation, local transportation, and creating the schedule must have been very difficult, but the CraftHub team did a great job, in my opinion.

Kevlin Henney opened the conference with the talk Non-Functional coding. His speech was fascinating and fun at the same time. Perfect start for the conference. I even brought a picture of one of the slides.

The Main Stage was full. You could even smell the scent of software development.

However, depression was not inside me this time, but it was pretty close to me as it was sitting right in front of me, as you can see in the picture above. I have no idea why the guy in front of me wore that T-shirt, but I was too shy to ask.

I used to love visiting meetups, as in addition to learning and networking, I always got a huge motivation boost for weeks. I am eagerly waiting for how long will the motivation boost last this time.

If you read my previous article, you know my plan was to visit Michael Feathers’, Dan North’s, and Ivett Ördög’s talk.

Michael’s talk was the first one on the list. So I went to platform 2 to listen to his talk. To my biggest surprise, there was a completely different speaker there. I suppose the talk was canceled, and I did not reload the schedule on my phone. Never mind. As a photographer, I loved Platform 2. Just look at the lights:

When I realized Dan North’s talk was a virtual talk, I decided to watch the talk from recording and visit a real-life talk instead, as that is the beauty of offline conferences. Listening to talks of top-notch developers from all around the world in person.

Luckily I could listen to Ivett’s talk, which was the last talk of the conference. She was talking about how to sell a big refactor to the management. I had expected a bit more practical advice, but the presentation was still interesting.

I found Fabrice Bernhard’s talk the most fascinating at the whole conference. I have to admit I am highly biased towards QA topics as I work as a test automation engineer. Fabrice was talking about the Dantotsu method with which you can drastically reduce the bugs in your code. I will write an article about his presentation as I would like to share the Dantotsu method with my team as I am pretty sure it would benefit us too.

To sum it up, there were many great talks during these two days, and I could only listen to a small portion of them thanks to the many parallel talks. I am pretty sure I will spend hours listening to the missed talks, and I will also watch some of the talks again as I would like to write a few articles about the ones I found most interesting.

Exhibitor Booths

Many of the conference supporters had a booth in the exhibition area. Epam, Morgan Stanley, Cloudera, Tesco, and of course CraftHub just to mention a few of them.

I believe the exhibitors knew most of the developers are there to learn and have fun and not to find a job, cause of this many of them prepared some minigames which made the booth visits quite memorable. Instead of job ads, they asked for your approval to get your contact info, so they could reach us after the conference with relevant content. I already got a few emails and I am sure more is on the way.

I have visited almost all the booths and played most of the games. You could win prizes by playing. Let’s mention the most memorable minigames.

Minigames

Epam had a smartphone slide challenge where you got points depending on how close you could slide your phone to the center of the target. Sliding the phone to the center gave access to the highest gift category. My first slide was too weak and the second one almost blocked the center, but I nailed it on the third try. Here is a pic of the game:

At adnovum, you had to play an AR game. You had to catch bugs using a phone’s camera on which either a cake or a bug appeared when you moved your camera above a symbol on the wheel. It was quite fun so I played it both on the first and second day too.

At the Tresorit booth, you had to open a safe by getting the code of the safe by answering a few questions from which you could figure out the code. Of course, we opened it and we could even get some chocolate from the safe.

At the Tesco booth, you had to decrypt a message which pointed to a location. I couldn’t solve it right there but a cigarette later I went back with the solution.

(Almost) all the exhibitors had a grand prize. You could participate in the drawing by sharing your personal info and the winners were mostly selected randomly. Unfortunately, I had not won any of the grand prizes but I was not sad about that because of the swags.

Swag Here, Swag There, Swag Everywhere

The exhibitors prepared for the Craft Conf with an extreme amount of swag. I did not realize up till now how fun it is to collect swags. There very everything you could imagine. Pens, notebooks, socks, sunglasses, t-shirts, and even rubber ducks.

I have to admit I might get too excited and collected too much swag, but I had a good excuse. I have collected almost all the swags for my son. This is mainly true, but I also gave some of them to my wife and I also kept a few. Here is a pic of almost all the loot I collected at the conference. 😊

The Staff

The CraftHub staff was very kind and helpful and we could even get mango-flavored beer from them, so it was no question whether we visited them or not. At the location, there were many people with a Staff label on their T-shirts who were there to help us if we had any questions.

Despite I had a map I had no idea where the Innovation stage was so a friendly staff guided me there. Actually, it was right behind me but I did not realize it. 😊

After the second talk on the main stage, I went to the CraftHub booth and asked them to increase the volume on the speakers’ mic as thanks to sounds coming from the lively exhibition area we had a hard time listening to the talks from the back rows. They were quite helpful and I am almost sure they increased the volume, as, from the following presentation, I had no problem with it.

Networking Party

Unfortunately, I could not participate in the networking party as I had to go home to be with my son as my wife had an important appointment, but one of my friends was there and he told me it was great. Epam supported everyone with 4 beer coupon so the mood was guaranteed. Maybe next year I can visit the party too.

Being a Kid Again

Even at the age of 40, I still consider myself a kid, as I never grew up. Especially nowadays, when I play a lot with my son I can enjoy such activities as playing lego, video games and much more without feeling any guilt of not spending my time on “useful things”. Luckily there were many activities at the Craft Conf which brought back my childhood memories.

Dodgem

I love dodgem since my childhood and for decades I never sat in a dodgem car as I thought that was for kids. Last summer and also recently I had the opportunity to realize how enjoyable it is even for an adult. I was even able to persuade two of my friends to jump into a dodgem. It was really fun.

Mini Train

As I mentioned before, the conference took place at the Hungarian Railway Museum. They have built a mini railroad at the location on which people could sit. I did not try this activity, but I saw many of the participants sitting on it. I brought you a picture to help you imagine it.

Candy Floss

I loved candy floss as a kid and I realized I love candy floss as an adult too. Luckily there was a candy floss stand right beside Cloudera’s stand so it was no question whether I tried it or not. Look at my happy face. 😊

VR game

There was a stand beside SAP where you could play Playstation games with a VR headset. As I had never tried it before, I took the opportunity. It was fun, especially as you had two handles in your hand with which you could interact with the virtual world instead of a mouse or a controller. As I learned, you can borrow these VR headsets if you have an event like a birthday party or want to have fun with your friends.

Final Thoughts

I would like to say thank you to Anna, Felhő bácsi, Médea, Károly, Barna, Vanda, Viktória, Anna, Renáta, Fanni, Bianka, Réka, Kata, Barbara, Veronika, Vivien, Marcel, Martin, Emilía, Kristóf, Gergő, László who are the people behind CraftHub the organizer of the Craft Conf. They made my first offline conference really memorable and I am pretty sure the organizers of my next conference will have a hard time outshining the Craft Conf 2022.

I would also say thank you to my friends Szabi, Fűrész, Karesz, and Laci who joined me personally for the time of the conference. I really enjoyed this 2 days and I just hope we can repeat it next year too.

A My Report About The Craft Conf 2022 Software Development Conference bejegyzés először CraftHub-én jelent meg.

Read through the newest article on the CraftHub website from Christian Posta who is going to be a speaker at this year’s Craft Conference in May. One of the biggest tech conferences in Europe is waiting for you in the heart of Budapest.

eBPF for Service Mesh? Yes, but Envoy Proxy is here to stay

Our goal here at Solo.io is to bring valuable solutions to our customers around application networking and service connectivity. Back in October, we announced our plans to enhance our enterprise service-mesh product (Gloo Mesh Enterprise) with eBPF to optimize the functionality around networking, observability, and security. To what extent can eBPF play a role in a service mesh? How does the role of the service proxy change?  In this blog we will dig into the role of eBPF for a service mesh data plane and what are some of the tradeoffs of various data-plane architectures.

Goodbye to the service proxy?

A service mesh provides complex application-networking behaviors for services such as service discovery, traffic routing, resilience (timeout/retry/circuit breaking), authentication/authorization, observability (logging/metrics/tracing) and more. Can we rewrite all of this functionality into the Kernel with eBPF?

The short answer: this would be quite difficult and may not be the right approach. eBPF is an event-handler model that has some constraints around how it runs. You can think of the eBPF model as “functions as a service” for the Kernel. For example, eBPF execution paths must be fully known and verified before safely executing in the Kernel. eBPF programs cannot have arbitrary loops where the verifier will not know when the program will stop execution. In short, eBPF is Turing incomplete.

Layer 7 handling (like various protocol codecs, retries, header manipulations, etc) can be very complex to implement in eBPF alone and without better native support from the Kernel. Maybe this support comes, but that is likely years off and wouldn’t be available on older versions. In many ways, eBPF is ideal for O(1) complexity (such as inspecting a packet, manipulating some bits, and sending it on its way). Implementing complex protocols like HTTP/2 and gRPC can be O(n) complexity and very difficult to debug. So where could these L7 functionalities reside?

Envoy proxy has become the de-facto proxy for service mesh implementations and has very good support for Layer 7 capabilities that most of our customers need. Although eBPF and the Kernel can be used to improve the execution of the network (short circuiting optimal paths, offloading TLS/mTLS, observability collection, etc), complex protocol negotiations, parsing, and user-extensions can remain in user space. For the complexities of Layer 7, Envoy remains the data plane for the service mesh. 

One shared proxy vs sidecar proxies?

Another consideration when attempting to optimize the data path for a service mesh is whether to run a sidecar per workload or to use a single, shared proxy per node. For example when running massive clusters with hundreds of pods and thousands of nodes, a shared-proxy model can deliver optimizations around memory and configuration overhead. But is this the right approach for everyone? Absolutely not. For many enterprise users, some memory overhead is worth the better tenancy and workload isolation gains with sidecar proxies.

Both architectures come with their benefits and tradeoffs around memory and networking overhead, tenancy, operations, and simplicity, and both can equally benefit from eBPF-based optimization. These two are not the only architectures, however. Let’s dig into the options we have along the following dimensions:

• Memory / CPU overhead – Configuring the routing and cluster details for an L7 proxy consists of proxy-specific configurations which can be verbose; the more services with which a particular workload needs to communicate, the more configurations it will need.
• Feature isolation – Applications are finicky and tend to need per-workload optimizations of connection pools, socket buffers, retry semantics/budgets, external-auth, and rate limiting. We see a lot of need for customizing the data path which is why we’ve introduced Wasm extensions. Debugging these features and behaviors also becomes demanding. We need to figure out a way to isolate these features between workloads.
• Security granularity – A big part of the zero-trust philosophy is to establish trust to peers at runtime based on current context; scoping these trust boundaries as small as possible is usually desirable.
• Upgrade impact – A service mesh is incredibly important infrastructure since it’s on the request path; we need to have very controlled upgrades of service-mesh data-plane components to minimize outages

Let’s look at four possible architectures where eBPF is used to optimize and short-circuit the network paths and leverage Envoy proxy for Layer 7 capabilities. For each architecture, we evaluate the benefits and tradeoffs of where to run the Layer 7 proxy along the lines of overhead, isolation, security, and upgrades.

Sidecar proxy (service proxy)

In this model, we deploy a sidecar proxy with each application instance. The sidecar has all of the configurations it needs to route traffic on behalf of the workload and can be tailored to the workload.

With many workloads and proxies, this configuration is duplicated across workload instances and can present a “sub-optimal” amount of resource overhead. 

This model does give the best feature isolation to reduce the blast radius of any noisy neighbors. Misconfigured or app-specific buffers/connection-pooling/timeouts are isolated to a specific workload. Extensions using Lua or Wasm (that could potentially take down a proxy) are also constrained to specific workloads. 

From a security perspective, we originate and terminate connections directly with the applications. We can use the mTLS capabilities of the service mesh to prove the identity of the services on both ends of the connections scoped down to the level of the application process. We can then write fine-grained authorization policies based on this identity. Another benefit of this model comes if a single proxy ends up victim to an attacker, the compromised proxy is isolated to a specific workload; the blast radius is limited. On the downside, however, since sidecars must be deployed with the workload, there is the possibility that a workload opts not to inject the sidecar, or worse, finds a way to work around the sidecar. 

Lastly, in this model, upgrades can be done per workload and follow a canary approach that affects only specific workloads. For example, we can upgrade Pod A’s data plane to a new version without affecting any of the other workloads on the node. The downside to this is injecting the sidecar is still tricky and if there are changes between versions, it could affect the app instance.

Shared proxy per node

The shared-proxy per node introduces optimizations that make sense for large clusters where memory overhead is a top concern and amortization of the cost of the memory is desirable. In this model, instead of each sidecar proxy configured with the routing and clusters needed to route traffic, that configuration is shared across all workloads on a node in a single proxy. 

From a feature isolation perspective, you end up trying to solve all of the concerns for all of the workload instances in one process (one Envoy proxy) and this can have drawbacks. For example, could application configurations across multiple apps conflict with each other or have offsetting behaviors in the proxy? Can you safely load secrets or private-keys that must be separated for regulatory reasons? Can you deploy Wasm extensions without the risk of affecting the behavior of the proxy for other applications? Sharing a single proxy for a bunch of applications has isolation concerns that could potentially be better solved with separate processes/proxies. 

Security boundaries also become shared in the shared-proxy per-node model. For example, workload identity is now handled at the node level and not the actual workload. What happens for the “last mile” between the proxy and the workload? Or worse, what happens if a shared proxy representing multiple workload identities (hundreds?) is compromised? 

Lasly, upgrading a shared proxy per node could affect all of the workloads on the node if the upgrade has issues such as version conflicts, configuration conflicts, or extension incompatibilities. Any time shared infrastructure handling application requests is upgraded, care must be taken. On the plus side, upgrading a shared-node proxy does not have to account for any of the complexities of injecting a sidecar.

Shared proxy per service account (per node)

Instead of using a single shared proxy for the whole node, we can isolate proxies to a specific service account per node. In this model, we deploy a “shared proxy” per service account/identity, and any workload under that service account/identity uses that proxy. We can avoid some of the complexities of injecting a sidecar with this model.

This model tries to save memory in scenarios where multiple instances of the same identity are present on a single node and maintains some level of feature and noisy-neighbor isolation. This model has the same advantages of a sidecar for workload identity, however it does come with the drawbacks of a shared proxy: what happens to the last mile connections? How is authentication established all the way back to the workload instance? One thing we can do to improve this model is use a smaller “micro proxy” that lives with the application workload instances that can facilitate end-to-end mTLS down to the instance level. Let’s see that in the next pattern. 

Shared remote proxy with micro proxy

In this model, a smaller, lightweight “micro proxy” (uProxy) that handles only mTLS (no L7 policies, smaller attack surface) is deployed as a sidecar with the workload instances. When Layer 7 policies need to be applied, traffic is directed from the workload instance through the Layer 7 (Envoy) proxy. The Layer 7 proxy can run as a shared-node proxy, per-serviceaccount, or even a remote proxy. This model also allows completely bypassing the Layer 7 proxy when those policies may not be needed (but keeping mTLS origination/negotiation/termination with the application instances.

This model reduces the configuration overhead of Layer 7 policies you see in sidecars but could introduce more hops. These hops may (or may not) contribute to more call latency. It’s possible that, for some calls, the L7 proxy is not even in the data path which would improve call latency. 

This model combines the sidecar proxy benefits of feature isolation and security since the uProxy is still deployed with the workload instances. 

From an upgrade standpoint, we can update the L7 proxy transparently to the application, however we now have more moving pieces. We need to also coordinate the upgrade of the uProxy which has some of the same drawbacks as the sidecar architecture we discussed as the first pattern. 

Parting thoughts

As discussed in “The truth about the service mesh data plane” back at Service Mesh Con 2019, architectures representing the data plane can vary and have different tradeoffs. At Solo.io, we see eBPF as a powerful way to optimize the service mesh, and we see Envoy proxy as the cornerstone of the data plane. Working with our many customers (of various sizes, including some of largest deployments of service mesh in the world), we are in a unique position to help balance the tradeoffs between optimizations, features, extensibility, debuggability and user experience. 

You can read more from the author on solo.io or check out the original article here.

A eBPF for Service Mesh? Yes, but Envoy Proxy is here to stay bejegyzés először CraftHub-én jelent meg.