Winning Over Developers

One of my favorite memories of living through Microsoft's adoption of secure development was sitting in a hacking demo by Foundstone for a lot of the Windows development team. The network security team were invited over to watch as well. The Foundstone guys demonstrated hacking an unpatched Windows 2000 system. As they got in to one part of the system I watched a developer hang his head and say "I wrote that code." I think that moment probably did more for software security with that developer than any other single thing. From that moment on he knew security features did not make code secure. He was won over.

During my SAIC days, all our customers were developing software for DoD. One of the fun parts of being a security guy in a DoD world is the reaction of people when you arrive. There are so many IA guys (DoDspeak for security folks, IA stands for Information Assurance) who think their job is to always say no that security guys who arrive to help are viewed with suspicion until they show they are different. Showing that could be a lot of fun.

Early on one of the teams we were called in to help were not too happy to see us. In the initial meeting, the dev lead sat in the back corner, leaning back with his arms crossed. He clearly didn't want us to be there. The team happened to be wrestling with a memory leak they could not find and we were able to find it via the static analysis tool we were using at the time. Solving that problem for them made them more receptive but not welcoming yet. As we passed on the results, and there were a lot since there was a lot of legacy C/C++ code that had been around for a long time, things looked grim until we set their priorities and waved them off worrying about fixing everything right away. Since their schedule permitted it, we sent them off on some easy-to-fix low-hanging fruit that allowed a very large, quick win. We then sang their praises to their boss. From then on out we were greeted pretty well as we looked at all their changes periodically every year.

Sometimes you have to do some work to convince developers even when they are on board. We had a team that had become convinced of the need for secure development just through hallway conversations with us. When we finally had a chance to review their code, it was actually in pretty good shape from a developer's perspective. They were the first app I had ever reviewed that had done every single database query right and had no SQL Injection issues we could find. Their app was a web app and they had done a pretty good job trying to prevent cross-site scripting by the proper type encoding of everything they thought was user input from their webpages. The problem was that they were thinking like a developer and not an attacker. It took demonstrating modifying the HTML on the client side to add things to their drop-down lists and firing up web testing/hacking tools like TamperIE (or Tamper Data if you are not stuck with IE) and Fiddler to show how much really was under the attacker's control. They quickly learned they couldn't think like developers to secure their code.

Most of the time you've got to tailor the approach to the people you are working with but in general:

• Make sure they know you are there to help them, not stop them

• Do not just identify issues, help show them how to fix them

• Never call their baby ugly

• Work with them to verify issues if you cannot test directly - they may know of mitigations elsewhere in the code or environment that you may have missed

• Realize every security bug found may mean a feature not shipped or a delayed delivery

  • Prioritize fixes

  • Identify other mitigations that may let them push a fix to a future release

• If possible, delay reports to their bosses until after they have begun fixes to praise progress

• If their schedule permits, go for big psychological wins even if the issues are not as serious as some others - knocking off a large number of issues quickly is very motivating

• If they have recurring issues (and most will) suggest training on those specific issues

Anybody have fun stories of winning over developers during security efforts or suggestion that they have found work well?