Category: Uncategorized

Lightweight Trust Building

I was speaking with some colleagues the other day about some low trust interactions that I had recently observed. Based on my description of my observations, one colleague commented that the interactions appeared to arise from a negative feedback loop that promoted a low trust environment between senior leadership, managers, and employees. To break that feedback loop, he recommended having the different groups take a Speed of Trust course. Having gone through a course myself, I agreed that it would encourage trust-building behavior, yet I was concerned that convincing even one group to take the course would be a monumental task. Another colleague acknowledged my concern and the difficulty with saying "Hey, I noticed you're in a low trust environment, would you like to take this trust course?"

This comment about the difficulty of opening a conversation on trust in a low trust environment prompted me to pick up the book Agile Conversations, which I had just started reading. Divided into two parts, the book describes a lightweight method of conversational analysis and the theory behind how it works followed by using the analysis in 5 kinds of conversations. The purpose of the book is to encourage high-trust, psychologically safe conversations regarding business value creation that enable participants to jointly set a direction for a project, commit to goals that will achieve it, and be accountable for the steps taken to achieve those goals. Intrigued, I kept reading.

The analysis of a conversation revolves around writing down a conversation and then comparing what was said with what was thought to improve the next conversation. A few good questions to ask during the comparison are:

  • When I ask questions, are they genuine questions or leading questions?
  • If I had difficulties or concerns with what the other person said, did I mention them and show a willingness to work with him/her to overcome them?
  • Am I being consistent in how I talk and act so that others can trust me?

These stuck out to me, as they ensured I stayed honest with myself about my intent (similar to the Speed of Trust) and promoted curiosity and transparency in my conversations. After determining what could be improved, what was said is revised to address the discovered problems. Then the revised conversation can be role-played to practice the improvements before the next conversation.

After explaining the analysis, the 5 conversations that followed were:

  • The Trust conversation
  • The Fear conversation
  • The Why conversation
  • The Commitment conversation
  • The Accountability conversation

Each of the conversations naturally leads to the next.

The Trust conversation ensures that all parties in a conversation are working from the same story of the problem or need that they are dealing with. When a person starts telling a conflicting story, follow the Ladder of Inference to determine what is causing the conflict so that it can be resolved. This ensures that everyone is working from a shared perception of the world and gives a common language that people can use to be understood. Once people trust that they will be understood, then they will more readily and easily bring up concerns that they have, leading to the Fear conversation.

As an aside, the Ladder of Inference is very close to the Catholic philosophical tradition of how we perceive the world and choose our actions based on those perceptions. Here is a rough sketch: The senses perceive the world, the imagination creates an image from that perception, the intellect recognizes the image, recognition prompts a value judgment which can trigger the emotions. Reason, reflecting upon the image, the value judgment, and the emotional reaction, determines the potential responses to the perception. The will makes actual one of the potential responses. For a more complete and accurate summary, see Aquinas' Summa Theologiae Book I. Questions 79-89 and 93

The Fear conversation provides an opportunity for the team to identify problems they fear will prevent project completion. Often the best place to start uncovering these fears is by examining the deviations of the team from their espoused way of working. These deviations are often disparate mitigations that create conflict within the team and prevent them from being effective. Once the problems are brought to light, the team as a whole can design mitigations for them. As it is probable that there are multiple causes for each problem, spend some time talking through multiple scenarios to make sure that the mitigation will be effective. As some mitigations may require an entire project to be effective, the team will start questioning if the current project is the most important thing to be done, leading to the Why Conversation.

The Why conversation builds a shared purpose and direction for the current project between team members and leadership. Within a timebox, each person has an opportunity to discuss the goals of the project and advocate for their position on them. To achieve a shared ownership of the project, it is crucial for the team to share and understand the interests that lead to the different positions, otherwise the conversation can result in an endless loop of debate. When the interests are visible, the team will naturally try to set goals that take them into account. As long as the final decisions regarding the project direction sufficiently address the different interests, the team will have a sense of ownership for the project, leading to the commitment conversation.

The Commitment conversation determines what project goal is going to be delivered and when. This requires that everyone involved in the project agrees on what is being delivered. This is only possible when they trust each other, have identified and determined sufficient mitigations for any concerns, and understand why this particular increment is the most important thing to deliver. Without achieving these first, people may be showing up, but they won't be fully participating in delivering the goal. Once the team commits to the desired goal, then work on the project beings, leading to the Accountability conversation.

The Accountability conversation communicates the progress of a project increment. It provides an opportunity for the team members to "radiate intent" and to receive feedback. This conversation is more about each person giving an account for their actions rather than being kept accountable for them. The structure of such a conversation can closely mirror the textbook standup questions: What is the current state of the project? What is planned next and the expected outcomes? what are the existing or upcoming obstacles? This conversation allows the project plan to adapt to new business constraints or missed user needs. It also creates a positive feedback loop that reinforces trust as conflicts are resolved through understandable compromises or the discovery of a more fundamental value for the business.

These 5 conversations and the analysis process looked very promising for overcoming the problems that I was facing. What made them immediately usable was that the first three conversations are all triggered when a conflict is perceived. However, the reasons describing why this analysis works gave me some cause for concern, which I'll be examining in another blog post.

What do you think? Is there possibly an even easier way to start improving trust within an organization? Are there additional conversations that could be added to this list? Is there a situation where taking a Speed of Trust course would be more beneficial? Let me know either in the comments or you can reach out to me on the various platforms where I'll be posting this.

My VPN dropped…again

During the COVID-19 stay-at-home order, I've been blessed to be able to work from home. However, I faced what many people suffered when switching from working in the office to working at home: a poor VPN connection. In my case, my connection would hiccup every 5 minutes.

This problem had shown up before we were forced to stay-at-home and I had worked on it with an excellent network administrator at my client. When tethered to my phone or using the office guest network, the VPN was rock-solid. It seemed to only fail on my home internet connection. This knowledge - combined with the stay-at-home order - gave me an excellent excuse to tinker on my home network.

The original network was a fairly typical setup:

graph LR; FTTH-->Modem[ISP Router]; Modem-->Laptop;

My first suspect in this setup was the ISP Router. Looking through its settings, I saw that passing through the public IP to another device would bypass most of the network stack in the ISP Router. As I didn't want yet another all-in-one router sitting next to the ISP Router, I looked for a vendor that supported a modular, easy to expand and maintain setup. I ended up choosing Ubiquiti Networks because of their unified dashboard that I could host myself and their "pro-sumer" hardware. A few days later, I brought home an EdgeRouter X (ER-X) and a Unifi AP Lite (AP) from the local Micro Center.

After a few hours of tinkering, I had the following setup:

graph LR; FTTH-->Modem[ISP Router]; Modem-->Router[ER-X]; Router-->AP AP-->Laptop

With this in place, the VPN drops stopped occurring and the latency dropped by a few milliseconds. I was elated, but then I noticed another problem. While the VPN remained connected, random latency spikes would cause pages to time out and video calls to drop. Ping would initially report dropped packets until the latency dropped, at which point the missing return packets would all simultaneously appear. My initial thought was that the packets were leaving the network but not making it back until the latency dropped. Suspecting that the IP passthrough hadn't really solved the problem, I attempted to physically bypass the ISP Router with the ER-X.

The ISP network will refuse service to unauthenticated devices attached to it. This meant that the ER-X needed to somehow authenticate itself in the same way that the ISP Router did. After some more research, I found three ways to accomplish this:

  1. Bridge the FTTH port and the ISP Router port on the ER-X
  2. set up eap_proxy on the ER-X and use the ISP router solely for authentication
  3. extract the authentication keys from the ISP Router and serve them from the ER-X using wpa_supplicant.

I picked the second option as bridging meant a slower connection and extracting authentication keys seemed legally dubious. I changed the physical configuration again and began configuring the ER-X.

graph LR; FTTH-->Router[ER-X]; Router-->Modem[ISP Router]; Router-->AP AP-->Laptop

I was able to get eap_proxy started only after I found an issue on the eap_proxy Github project and downgraded the firmware on the ER-X to an older supported version. However, the auth packets from the ISP network still weren't making it to the ISP Router. After tweaking different settings and some more research, I found a guide on Github which solved my problem. To make sure that only the required setting were in place, I reset the ER-X to it's factory settings and then walked through authenticating with the ISP network one last time.

With this final set up, I again tested my VPN connection...no dice. Armed with the additional details now in the Ubiquiti dashboard, my new suspect was the AP, as I saw a jump of WIFI retries when I was working in my study. So I bought an Ethernet cable and ran it to the study, to see if that would solve the problem...no dice. I replace the cable between the ER-X and the FTTH connection...no dice.

I was rapidly running out of options. Mildly frustrated that my hardware fixes didn't make any difference, I started tcpdump on the router and watched the VPN keepalive packets between the laptop and the VPN server. Then my understanding of the problem flipped. The laptop would pause in sending the keepalive and then it would send a whole bunch, matching the latency spike. The laptop itself was source of the problem. With this new insight, I set out to show whether it was a hardware or software problem.

To test whether it was a hardware problem, I set up a Linux VM on the laptop and connected to the VPN within the VM using openconnect. No problems. Perhaps the supplied AnyConnect client was the problem? I looked through the brew repository and discovered that an openconnect client existed for macos. I immediately installed and tested it. no problems.

Reaching this resolution took a couple weeks and it's held up for several weeks now. The open-source alternative to the AnyConnect client has been working just fine. At some point, I'll need to re-install the AnyConnect client to see if there was some old driver or library that was causing the network problem. It's been very nice to be able to get my work done without my connection dropping all the time.

Where did this 404 come from?

Recently, a friend reached out to me to help him debug a 404 error that didn't make any sense in his Kubernetes cluster.

He had a rather simple setup:

  • a pod for his application
  • a nginx ingress used as a load balancer
  • a nginx ingress that pointed to the application pod
  • cert-manager for automatically generating certificates from Let's Encrypt

After talking through his setup, I put together the following diagram:

Mental model of his setup

Rather than the 404 error, he expected a 308 redirect from the root domain to the www subdomain, e.g. https://example.com redirecting to https://www.example.com.

With this information, I traced the 404 through the logs to the ingress. So I cracked open the existing ingress configuration: 

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  namespace: my-app-ns
  annotations:
    nginx.ingress.kubernetes.io/from-to-www-redirect: true
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - backend:
          serviceName: my-app
          servicePort: 80
        path: /
      - backend:
          serviceName: cm-acme-http-solver-4k39l
          servicePort: 8089
        path: /.well-known/acme-challenge/d34db33fde4db3ef
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-7l45p
          servicePort: 8089
        path: /.well-known/acme-challenge/d3adb33fd34dbe3f
  tls:
  - hosts:
    - www.example.com
    - example.com
    secretName: my-ingress-cert

Within this configuration, I noticed 2 things:

  1. The nginx ingress annotation which sets up the www redirect.
  2. The "/.well-known/acme-challenge/" paths indicated that the cert-manager had been set up to use the HTTP01 Challenge for Let's Encrypt

Perhaps the host stanza for "example.com" was the problem? I removed it and applied the configuration to the cluster, watching the logs. Once the configuration was applied, cert-manager stepped in and updated the configuration to re-add the missing stanza, at which point the nginx-ingress refused to create the redirect to the www subdomain. There was the problem!

I spoke with my friend about switching cert-manager to use the Let's Encrypt DNS01 Challenge. He pointed out that cert-manager didn't have an integration for his DNS provider, which is why he hadn't used it in the first place. Until that integration existed, we needed a solution that kept the existing cert-manager configuration and didn't use the "from-to-www-redirect" annotation.

After some trial and error, and some help from Stack Overflow and the DigitalOcean community, we ended up using the following annotation: 

nginx.ingress.kubernetes.io/configuration-snippet: |
  if ($host = "example.com") {
      set $test "1";
  }
  if ($request_uri !~* "^/.well-known/") {
      set $test "${test}1";
  }
  if ($test = "11") {
      return 302 https://www.example.com$request_uri;
  }

Because nginx does not allow the if directive to handle multiple conditions, we had to use three conditions. The first two test the actual properties we wanted to check, while the third tests the outcome of those two previous checks. This workaround was rather gnarly yet ensured that cert-manager could renew certificates for the root domain while redirecting all other traffic to the www subdomain.

Those 404 errors

When I posted last, I noticed that the nice permalink took me to a 404 error page rather than my intended blog post. So I temporarily disabled the nice permalink until I could debug the problem.

After digging into it, it turned out that I was missing this stanza from my Apache configuration: 

<Directory /var/www/wordpress/>
    AllowOverride All
</Directory>

This meant that Apache was ignoring the .htaccess file that WordPress generated for permalinks.

Once I put that stanza in place, the permalinks started working again.

Many thanks to author of a DigitalOcean Community Tutorial for helping me solve this problem.

EDIT: Turns out, the post on LinkedIn was still returning a 404, since it wasn't prefixing the link with https://. Now I've setup a permanent redirect with:

Redirect permanent / https://jonathancamara.com/

Let’s try this again.

Much has happened since I last posted something: a move, a new job, kids, and more! Reflecting on my motivations for writing, I realize that I was attempting to write at length on topics that I knew next to nothing about. With this in mind I've thrown out all my drafts and plan to keep my posts short and simple.

Reflecting on recent events: The crisis in the Catholic Church, the increasingly extreme divisive language in politics, and the spiritual and moral poverty in society, I've realized that so much of what we argue about is peanuts in comparison to God's plan. I'm hoping that these blog posts are a kind of almsgiving.

We all need to turn to the Lord God, beg for his forgiveness and mercy, and seek to conform our wills to His. Only then will the Church recover, the politicians forgive, and society be spiritually and morally enriched.

GO TO CONFESSION.

For the men who read this,  I invite you to come to the Holy League Holy Hour for Men at Annunciation Catholic Church on Wednesday, March 13. Church doors open at 5:00AM with Exposition of the Blessed Sacrament and the Sacrament of Reconciliation beginning at 5:30AM, followed by Mass at 6:30AM, and fellowship with other men from around the Archdiocese of Galveston-Houston after Mass.

Thoughts about Apple’s Customer Letter

I've done some research on the court battle between Apple and the FBI (AKA read some blogs, listened to coworkers) and the Apple's letter to customers does a fairly decent job in explaining the current situation between them and the FBI. However, the letter glosses over some of the underlying political, technical and philosophical problems that I believe are essential to understanding why Apple is refusing to comply with the FBI's demands.

Politically, this is 'dynamite' for the case the FBI has been making for the last several months for creating a backdoor in encryption. Up to this point, the FBI has made the case using the media with the goal of having state or federal laws passed to create this backdoor. This case gives them the ability to 'speed up' the process by having this decision made in the courts, rather than having to slowly work through an increasingly stagnant legislative system. There is also plenty of evidence that this is an attempt by the FBI to make an example out of Apple. If the FBI can strong-arm what is currently the most profitable tech company in the world - a company that markets its products based on quality and security - into creating a backdoor into its products, then all the other tech companies will more likely 'roll-over' and put in back-doors as well. Also, we have evidence from several whistle-blowers and tech companies that the FBI is known to intentionally modify technical equipment to perform surveillance, so for them to go ahead and create the backdoor themselves for this particular iPhone is not exactly outside of their capabilities. In addition, Apple specifically requested that the FBI not go public with this case, but the FBI did so anyway, which is probably one of the reasons why the customer letter was written.

Technically, a backdoor in encryption is just a flat-out bad idea. Encryption is used for online shopping, online banking, healthcare records, and government secrets, just to name a few. In the case of the iPhone, this encryption is used to help ensure that if someone steals your phone, and you have it locked, then there is no way for them to access the data on the device. It is also used to ensure that messages sent between phones cannot be intercepted and modified between the sender and the receiver. The obvious downside is that it also can prevent law-enforcement from legitimately accessing the phone of a convicted criminal. However the idea of developing encryption that is strong for government but weak for individuals is flawed. In fact, there have been a number of recent computer attacks that happened because certain security technologies which were compliant with the now rescinded rules from the 80's and 90's against exporting encryption technology. The best analogy that I can think of putting a backdoor in encryption is the scene in the Lord of the Rings movie where Denethor says the Ring "should have been brought back to the citadel to be kept save, to be hidden dark and deep in the vaults, not to be used, unless at the uttermost end of need." The FBI would be creating something in order to be "safe", and yet would set up the very weakness that terrorist organizations and other bad actors would love to exploit, as there would be no way to stop them, since the hole in our defensive structure would be there by design.

Philosophically, the whole question is being placed by the FBI on the false dilemma of 'security versus privacy'. Apple is attempting to shift the conversation to more 'freedom versus fear'. The idea of 'security versus privacy' is a false dilemma because security is impossible without privacy and vice-versa. In order to establish the common good, respect for the person requires that his privacy be safeguarded, and that the security of the community is maintained, (see Catechism of the Catholic Church 1907-1908) To sacrifice one for the other would be destructive to the common good and would prevent either from being realized at all. Apple's attempt to re-frame the question is definitely closer to what we should be concerned about, but I do acknowledge that while they say the right words, the meaning behind them is very different, being rooted in a more individualistic sense of freedom, which I talk about a little bit here and here.

For a more in-depth look at this story, I highly recommend Troy Hunt's article: Everything you need to know about the Apple versus FBI case.

Personal Responsibility vs. The Tribe(tm)

In my last post, I established the difficulty of being personally responsible without accountability to something or someone, and hinted that perhaps the tribe should be that someone.

Assuming that my hint was true, when a person would act, he would be responsible to the tribe he lives in. This would lead to the building of trust between the members and potentially to benefits for the entire group that would be impossible for a single person to realize by himself. It would appear that within such a community, a person would truly be able to realize his full potential.

However, wouldn't this land us back in the problem we wanted to get out of in the first place? The supposed trust and benefits could also lead to a kind of group-think that would smother a person's self-agency. Have we not seen societies that suppress the individual in order to maintain the status quo? We have heard the consequences of living in such a group when its members excuse their actions saying, "I was just following orders."

This leaves us in an apparent conundrum, for an individual cannot be a law unto himself, and yet if he is part of some tribe, it will eventually smother his ability to make his own decisions and be personally responsible.

But this very conundrum is based on a flawed understanding of society and how it relates to man, for if society did eventually strip man of his self-agency and personal responsibility, then society, on a scale larger than a few families, would never have formed in the first place. Man must have some need for society to exist for, as the ancient philosopher Aristotle stated, "Man is by nature a social animal." Furthermore, the human person could not exist without society, for it is into society that he is born.

The relation between the human person and society is very tightly knit, for society is necessarily composed of human beings, and we human beings cannot come into existence without the basic building block of society, the family. The goal of society is to accomplish the personal good of man by ensuring the common good of all. Thus it would appear that it is in man's best interest to take personal responsibility of those duties within society which he is capable of undertaking that will accomplish that common good. It would be irresponsible for any person and suicide for any society to neglect or undermine the accomplishment of those duties.

Bearing this in mind, let us return to the original statement: "People need to break from the tribe, embrace their personal agency, take responsibility for their actions, and, by chaining the darkness within, achieve self-actualization!"

Perhaps it would be more reasonable to say: "People need to recognize their duties towards others, take ownership of those duties, strive to overcome the selfishness that is destructive to themselves and society, and, through serving each other each day, achieve the actualization of themselves and all of society."

A New Year, a New Resolution

With the arrival of 2016, I've made a resolution to write 500 words once a week with some friends.

Sigh.

Now that that is off my chest, let us proceed with the first installment.

Recently, I was wondering about radical individualism and personal responsibility. My train of thought was started by an email thread from a group of coworkers. In the brief discussion, amid the glittering generalities, appeals to prestige, jargon and so forth, and with an overarching sense of radical individualism and a grumbling against any form of perceived oppression by a governing power, the same old line was trotted out: "People need to break from the tribe, embrace their personal agency, take responsibility for their actions, and, by chaining the darkness within, achieve self-actualization!"

Face-palm.

(There was more, which involved the words "shibboleths" and "fetishes", but properly going through it all would have made that sentence even longer.)

Just reacting to that statement and then passing over it could have been simple, but this is one of those statements that bothers me and I knew that I wouldn't rest until I had dealt with it in full...or at least as much as I could in these 500 words.

The fundamental question, at least as I see it, is this: Is it possible for such a person - one who has reclaimed his personal agency from the tribe - to be truly responsible?

Initially, the statement under consideration actually sounds great. If a person has somehow surrendered his personal agency, that is, his own ability to make decisions, to some larger group of people, then for him to take back that ability would seem to be a positive thing. If he subsequently took responsibility for exercising this regained ability, that would be even better. Such a person would refrain from excuses like "I was just following orders," or "The devil made me do it." All that he would have to do to start down this road to free choice and personal responsibility is to break from his tribe.

However, upon further consideration, this statement breaks down. In order for a person to be responsible, then there is something or someone that person must be responsible to. Now, it could be argued that the person is being responsible to himself alone, but then we are faced with a problem similar to Plato's Ring of Gyges. If a person is responsible only to himself, and not to others, then he could not be held accountable for his actions, good or bad, except by himself. But the statement we are considering explicitly states that there is a need to "chain the darkness within." If a person is a rule unto himself, how would he even know what darkness is and that it would need chaining, much less that it is within himself?

So, the accountability inherent in responsibility cannot be laid solely in the hands of a single human person, but then who or what else should this person be accountable to?

What about the tribe?

Overcoming Project Paralysis

I'm fighting passion project paralysis.

I want to work on an open source project, but I can't seem to find the energy to get over that initial inertia of choosing a project, setting up the environment, etc.

Whenever this happens, I start worrying: what happened to the passion that I used to have? Why do I not have a project that I just want to hack on? Am I just complacent with the software I have, since it pretty much accomplishes what I want it to do?

To overcome the paralysis, I sometimes play little games to get myself to just start: I'll use a random project picker that I wrote, and then work on the project that is picked for me; or I'll listen to a podcast and pick one of the problems that is brought up by one of the hosts and start working on it. But too often, I'll spend most of my time just getting my head around the project that was randomly picked, or I'll move on to the next podcast (cause you can never listen to too many, right?), and nothing gets done.

Sometimes, I find that procrastinating is more enjoyable...argh.

Sometimes, I remember one of the questions that always bother me when I am actually working. But that is the subject of a future post.

Pair Program with Me

Lately, I've been doing some research into popular programming practices that have apparently been around for years, but which I've never heard about before - such as pair programming. In pair programming, you and another developer team up to work on the same code and may, sometimes, share the same computer. Several studies have been done which support the benefits of pair programming, and how it helps improve both the code and the programmers. With this in mind, I've decided to take on the challenge placed on pairprogramwith.me and post on G+, Facebook, and Twitter that I am interested in pairing on open-source projects. I am definitely new to pair programming in general and still feel like a newbie when it comes to programming, though I can pass FizzBuzz. If you are interested, just click below and let me know so that we can schedule a time.

 

Pair program with me!