Separating the Sheep from the Goats with Political Data Mining

In the 2012 U.S. presidential election we witnessed an unprecedented usage of data mining to increase the effectiveness of political campaigning. The incumbent President Obama soundly defeated the challenger Romney, and the campaign team and pundits widely credited the Obama campaign’s strategic and highly integrated data mining tools with widening the gap.

ComputerWorld did a story yesterday (Obama and Romney big data experts continue the battle as businesses) on how Obama’s data analytics team had re-formulated itself as a business and was most recently seen helping Cory Booker win the NJ Democratic Senate primary. The article talks about how the team determined, down to the individual, the likelihood of each person participating in the primary election. It is no secret that Obama’s campaign team used such information to determine which voters received phone calls, visits, or perhaps even the tone of mailings they may have received.

But assessing the likelihood of any one individual’s participation in an election cycle is of limited value. Encouraging an apathetic voter who doesn’t see things with your worldview can easily backfire. The last thing you want to do is “get out the vote” for individuals who won’t support you. Reading between the lines, it is clear that the winning data mining team must be going beyond assessing participation and on to assessing the probable outcome, expected value if you will, of each person’s vote using models trained on income levels, age, race, zip code, and the like.

In essence, political data mining gives a campaign the ability to separate the sheep from the goats. The sheep, the masses moving along mindlessly with the objectives of the campaign, are assumed to be a known quantity, a give-me of sorts. The goats are the wildcards who ensure victory or defeat and require intensive manipulation. They are the margin by which elections are won or lost. They will also become the scapegoats on whom all blame for a party’s failure to win is placed, much as we saw with the mainstream Republican party blaming the conservative wings following the Romney loss.

There is something inherently offensive about this use of data mining. The known quantities, or sheep, aren’t spoken to and aren’t listened to. The goats get all the attention and become the focus of expert manipulators. When taken to its most extreme, you end up with the Kevin Costner film Swing Vote in which it becomes clear that neither side really believes in any of their core principles and will only say and do that which will get them the win. Governor Chris Christie of New Jersey publicly announced his support for such strategies yesterday as well (Chris Christie lays out argument for 2016).

Geek Speak, Rants

Using Switch-Case Statements and Fall Through Effectively

Switch-Case statements with fall through in Java are often cited as being the source of accidental bugs. Skip a ‘break’ statement and you end up processing the next case after the one you meant to process. Clearly there is a use case for fall through or it wouldn’t be a language feature. While coding something today, I remembered that it is great for processing hierarchical data types where each CASE value is logically a subset of another. I don’t think this is new information, but it was a helpful thing to recall for my own development. In the example below, I am processing dates. What happens to a SECOND should also happen to a MINUTE, what happens to a MINUTE should also happen to an HOUR, etc. I could repeat the code that applies to each case, but the fall through is much more succinct yet still understandable.

     /**
     * For a given date, find the previous value that is an
     * exact value in the value set of timeUnits
     */
    public static Date floor(final Date date, 
                             final TimeUnits timeUnits) {
        final GregorianCalendar calendar 
                = new GregorianCalendar();
        calendar.setTime(date);

        // fall through to each smaller unit
        switch (timeUnits) {
            case MONTH:
                calendar.set(Calendar.DAY_OF_MONTH, 1);
            case DAY:
                calendar.set(Calendar.HOUR_OF_DAY, 0);
            case HOUR:
                calendar.set(Calendar.MINUTE, 0);
            case MINUTE:
                calendar.set(Calendar.SECOND, 0);
                calendar.set(Calendar.MILLISECOND, 0);
                break;
            default:
                throw new IllegalArgumentException(
                        "Unexpected enum value: " 
                       + timeUnits.name());
        }

        return calendar.getTime();
    }

Geek Speak

Moving the Big Rocks

The world lost Stephen Covey recently. Although the cult-like admiration of some of his followers always weirded me out a little, I think he had some great lessons to teach us. His picturesque language describing things like “moving the big rocks” helps provide tactics for dealing with real-world challenges like prioritization. I moved some limestone around the yard this summer and, regrettably, didn’t find much applicability of that particular lesson, but it is one I have tried to apply to my work and side projects.

I’m actively working on a side project nearing its two year mark now. The subject, generally speaking, is textual data mining and predictive analytics. I’m having a blast, and hoping to turn a corner and launch my software to at least a limited audience this fall. At this point, I need to come up from programming for some air to get some ducks in a row on the business side. I’m expecting to have to build my brand recognition and a customer list, but based on my reading of the market, the demand for a general-purpose SaaS solution is there. Twitter reported in the past that as many as 60% of all Tweets are never read by a person. Social media networks, interactive forums, and internal business systems have a ton of user-generated textual content that isn’t leveraged much beyond waiting for someone to come across it in a search or file listing.

Some of the big rocks I’ve been moving are: defining a business problem, creating a software architecture to support a solution, prototyping workflows and user experience, researching industry challenges, analyzing SWOT (strengths, weaknesses, opportunities, and threats), and building my professional network. It’s been a tremendous learning opportunity and I’m definitely looking forward to seeing what’s next.

other

Reluctantly Facebooking

It’s hard to believe I haven’t posted since February. Given the topic of my last blog post and the length of time that has passed since, I feel like I should jot some thoughts down.

I reactivated my Facebook account when I realized in late May that it was, indeed, my only way to contact some people I actually did want to keep in touch with. There’s a lot of utility in the software, all its invasive advertising and privacy warts aside. To help keep my sanity, I have prioritized some user feeds for notifications in the Android application while generally avoiding logging into the web site. On the few occasions that a visit to the full Facebook site is in order, I have begun launching a Chrome incognito browser window to separate it from my other online activities. In exchange, I imagine at least that I am minimizing the invasiveness of online tracking features such as Facebook Connect. As an added bonus, maybe I moved down the priority of a government watch list somewhere. After all, Facebook Abstainers Could Be Labeled Suspicious. This is a weird new world we live in.

I’m re-reading a book from the mid-90s called Silicon Snake Oil by Clifford Stoll. I started reading it when I was a super-nerdy teenager who was convinced the Internet was the best thing since sliced bread. I don’t recall if I ever finished it the first time, as I found his questioning of everything I found so magical to be disconcerting. This time around, I find most of his assertions about the utility of technology to be laughable. He openly questioned the feasibility of social networking (without knowing that term), e-commerce, and a variety of other things that we take for granted now. What he was very spot-on about is the loss of “realness” that comes with so many online interactions. He painted beautiful pictures with his words that made library shelves and card catalogs, human interaction, and outdoor adventures seem charming and exciting rather than relics of bygone eras. As a whole, the book is so anachronistic now that it seems absurd, but I would be very interested to read a follow-up essay from Mr. Stoll on what we have lost as a society with the Internet.

other

Adieu to Facebook

I deactivated my Facebook account tonight. I’ve been threatening it a while now. I finally reached my tipping point.

While filling out the deactivation form, I was asked to select a reason. I selected ‘Privacy Concern’ and filled out the explanation below:

My privacy concern is that no one seems interested in having any. The oversharing on here pretty much tells me that our society is teetering over the edge of depravity. One of my Facebook friends (a parent of a small child) posted a comment on another parent’s picture posting of their kids bowel movement. With the new feed features, that of course “plopped” right into my news feed. I hope Mark Zuckerberg ends up broke and homeless for this foul invention.

Please stop the insanity, people.

Rants

Spring Annotations That Look More Like the Disease Than the Cure #Java

First off, I will preface this post with the caveat that I have a lot of respect for the Spring framework and the talented engineers who brought it to the community.  Given a blind choice between a Spring framework and a competing framework from another vendor, I think I would be much inclined to select Spring.  I understand the pressures that public companies are under and that they have to cater to silly forces and marketing campaigns to stay relevant and profitable.

I was watching a webinar today about changes in Spring Security 3.0/3.1, and I have to say that it provided me with one of the most readily explainable examples of what has been giving me uneasy feelings for some time now, namely the proliferation of Java annotations for imprudent uses.

Annotations are a great mechanism for providing cross-cutting functionality in a more readable and less invasive mechanism.  In many cases they enable such object-oriented principles as “favor composition over inheritance”.  Behaviors perpendicular to business logic such as logging, auditing, and transaction management can be indicated in a simple way which points to functionality defined and running elsewhere.  Some functionality is, in my humble opinion, clearly inappropriate in such a container.  For example, I find it a rather poor way to “externalize” configuration.  Your username, password, ports, and protocols for a 3rd party integration don’t belong in annotations.  Nor is there any value in defining algorithms or strategies in 20 lines of bizarre declarative annotations.  Code has its place as do annotations.

The specific example I saw in Spring Security that I am highlighting was the new annotations that support an expression language.  It is, in all actuality, the classic anti-pattern of a magic string.  Scripting code to support permission evaluations is written in yet-another-expression-language and placed directly alongside the code it influences.  The code is untyped, completely framework-proprietary, and brittle.  It doesn’t take much imagination or foresight to see the things that can go wrong, like a very simple refactoring not being reflected in annotation code.

So far I’ve been very abstract, so here’s what I’m talking about:

  @PreAuthorize("hasRole('ROLE_ADMIN') or hasRole('ROLE_EDITOR')")
  public void doPrivilegedStuff() {
     ...
  }

Users of dynamic languages who have fully bought into Test-Driven Development might not give this any thought. However, the reality of the Java applications I have worked with professionally is that unit testing is considered a luxury and 20-30% line test coverage is pretty remarkable. I do know this type of code (and it is CODE) hidden in a inline DSL is likely to suffer from typos and errors that are only caught at runtime. If you are especially unlucky, a typo might even make security silently allow access inappropriately.

I think a better approach would be something like:

  @PreAuthorize(AdminPermissionEvaluator.class)
  public void doPrivilegedStuff() {
     ...
  }

  public class AdminPermissionEvaluator implements PermissionEvaluator {

      boolean hasPermission(Authentication authentication, 
                            Object targetDomainObject, 
                            Object permission) {
          return isUserInRole(authentication, Roles.ADMIN) 
                  || isUserInRole(authentication, ROLES.EDITOR)
   }

  boolean hasPermission(Authentication authentication, 
                        Serializable targetId, 
                        String targetType, 
                        Object permission) {
      return isUserInRole(Roles.ADMIN) 
              || isUserInRole(Roles.EDITOR);
  }

  boolean isUserInRole(Authentication authentication, GrantedAuthority authority) {
      for(final GrantedAuthority authority : authentication.getAuthorities()) {
          if(authority.equals(authority)) {
              return true;
          }
      }
     return false;
  }


(This is example blog pseudo-code that has never been in an IDE or compiled, but you get the idea).

So, what did we give up and what did we gain? We gave up some brevity and completely inlining the permission evaluator. We gained: 1.) strongly-typed code, 2.) a reusable and testable permission evaluator, 3.) a guarantee that refactoring won’t trash our security, 4.) the ability to unit test and validate security at build time, 5.) death to magic strings, 6.) A separate class in which we could inject DAOs, external services, or integration code necessary to determine the specifics of fine-grained permissions.

My usage of Spring Security until now has basically consisted of HTTP basic authentication for web apps and REST services as well as basic role-based security where the roles are known and can be enumerated when the code is written. So, I am almost sure there is a way in Spring Security to do exactly what I am suggesting. What I take issue with is a framework purporting to be a better alternative to JavaEE providing functionality that will almost NEVER improve development and will probably actively harm a novice developer who uses it, especially in the context of long-term MAINTENANCE development. A framework should actively steer users toward smart solutions while still allowing them low-level access to interfaces, base classes, and configurations which can be harmful if used incorrectly. It shouldn’t just hand them the dangerous functionality as a heralded and celebrated feature.

It’s not just Spring Security going down this road either. Spring’s core functionality is starting to provide silly ways to do things with annotations also. Using Spring but without tainting every class in your application used to be something that the Spring community really preached. Now, everyone seems to be screaming “death to XML configuration” while taking 3 steps backwards in code maintainability. Just because you can make it fit into an annotation doesn’t mean you should. Hell, you could just mix metaphors and put a whole XML config file into the a String value of an annotation if you really want to make a screwed-up application.

I can’t help but think of flawed Consult-Use-Retire-Consult cycle that I have observed in organizations with insufficient critical thinking skills. A Consultant produces some software using some new “cutting edge” framework or technology, a client Uses it for a time and eventually finds it doesn’t meet their needs, a developer or support team is unable to make it work causing it to be Retired, and finally a Consultant is paid to create the same base functionality but with a larger feature set and a completely different design that will soon lead to another consultant starting from scratch. In other words, the only people that ever seem to profit from poorly maintainable code are hourly consultants and the companies that employ them!

In conclusion, I like Spring. It’s a cool set of frameworks that have done a lot of great things for Java developers. But, I can’t help but feel the organization may be starting to lose its way dumbing down development to the point of framework-endorsed worst practices. At the end of the day, good software usually isn’t all that sexy or easy and trying too hard to make it so usually points toward lack of maturity on the part of a development team.

Geek Speak

Ron Paul Making His Move?

The election results in New Hampshire tonight seem pretty unexpected to me.  CNN is projecting Ron Paul to take second while trailing Romney by 15 points.  I hate to read too much into it, but it seems like both the GOP establishment and the Tea Party are being abandoned.  I can’t help but hope the trend continues because I am adamant that neither group represents a clear path forward to for real solutions.

The GOP establishment, through the emphasis on Romney’s electability has made it clear that they want to put an R next to the President’s name with few other requirements.  The Tea Party talks a lot about reduced taxes and smaller government while also endorsing certain untenable and unconstitutional practices.  A Romney administration might well create and enlarge bureaucracies like the Obama health plan.  It is not inconceivable to me that a Tea Party administration might be equally large but with an emphasis on massive defense spending and beefed-up internal law enforcement to make sure all the subjects toe the line.

Ron Paul offers a new option.  His votes in Congress were based on one effective rubric–Constitutionality.  Oddly enough, that had him voting no on almost everything.  His critics call him naive and out of touch because he doesn’t think that the core governing principles required to sustain a free democracy have changed with the shifting winds of 200-odd years.  They think he is the naive one when he makes a cogent argument against a foreign policy based on obvious and covert manipulation of the internal affairs of countries around the globe.  Paul takes issue with turning our own country into an authoritarian police state to further special interests on the left or right.

Maybe this will finally be the year of the libertarian.  While I’m hoping, maybe the Occupiers who aren’t insane will take meaningful action by aligning behind Dr. Paul.  Real solutions don’t require governmment to right wrongs or punish the corporate interests, but rather to just get out of the way.  Greed and capitalism aren’t the answer but they aren’t the enemy either.  American ideals like courage, honor, and freedom embodied in our history and Constitution hold all the solutions we need.

other , ,

The Real Problem with Occupy Wherever #OWS

I have blogged before about what I see as zombie groups united more by rage than any particular ideals.  I think some aspects of what I said then could be applied to the Occupy groups, but I also recognize that most of the members of the Occupy movement have been somewhat better behaved than the mobs that committed sweeping acts of arson and destruction in London in August.  Regrettably, some have been just as bad, but I will give the Occupiers the benefit of the doubt and assume that was  not the wishes of the larger group.

What I have observed from both sides of Occupy debates, from the fiercest advocate to the most tenacious opponent, is that the groups seem to lack any shared context of cultural references, history, or even terminology.  It is truly frightening to me that we have groups in this country angry at each other who aren’t even discussing things from the same frame of reference.

Take political ideologies, for example.  The Occupiers have been bashing capitalism and their opponents have labeled the Occupiers as socialists.  Clearly then, socialism and capitalism must be opposites, right?  I disagree.  The opposite of any functional system of government is tyranny.  Socialism has worked for some, generally small, communities united by some ideal or purpose; religion or shared ethnic history has generally been that purpose rather than socialism itself.  (It is the elevation of socialism itself to the status of a great ideal or religion that makes it so appalling to some of us.)  Capitalism has brought great economic gains to many, and astonishing gains to a few, through a sometimes indiscernible blend of  corruption, ambition, and genius.  Though different in theory, in practice both socialism and capitalism have allowed power to concentrate into the hands of few, be it due to extraordinary service to the Party or concentration of wealth.  Power held by the few, unchecked, will always turn into tyranny.  What you call the group that holds the power is really irrelevant.  The fact that the Party is the only permitted ideological entity in a socialist country explains why socialist countries have always had a government collapse before any real change can occur.  A capitalistic, representative democracy like the U.S. was intended to be is rather built for change, and has been through many changes through the years.

Yes, the sway of money through lobbyists and other leverage is powerful, but the power of an IDEA is still highly valued by the people of our country almost universally.  How that idea is given legs can sometimes make all the difference between it being embraced or discarded.  Physically occupying a place where you are not wanted for prolonged periods of time simply makes you a disordered nuisance.  It is, in essence, a confession that your idea is so weak that it can not stand on its own.  A truly peaceful assembly of people with a shared idea and goals could appear en masse for a few short hours at strategic locations and be taken seriously.  An unorganized group of homeless dreamers camped out among their own filth in parks with no common ideology or demands will never be taken seriously.

Let’s look at property rights as another area where the Occupiers and their opponents seem to lack any common ground.  The Occupiers have stated that they believe the 1% unjustly hold that wealth, and so much or all of it should be taken and redistributed.  They are not protesting outside government offices, which makes it seem they do not wish this inequality to be rectified through the tax code, trials (in the case of the assertions of criminally-gained wealth), or any other channels our system of government offers.  Yet, knock down a few tents and throw a few of their possessions in the back of a trash truck, and they become livid at the affront to their “rights.”  They are willing to destroy the lives of the working rich by cutting off their businesses (the NYSE, the Oakland ports to name a couple) and deprive the even wealthier of generations of accumulated wealth, yet their right to occupy a park in perpetuity is sacrosanct.

Though power in a capitalistic country such as ours does show a concentration to the few in the form of wealth, true capitalism can only exist alongside freedom.  Capitalism does dissipate as fast or faster as freedom, and I believe this can be seen in both the recent bailouts (depriving business of the logical freedom of failure) and in expensive regulatory burdens to business such as government-mandated health insurance.  In some cases, the Occupiers are right in pointing out problems in our system, but what they would identify as the flaws of capitalism can be more accurately explained as the creeping effects of tyranny.  Governmental favors to private business (Halliburton, Solyndra?), bailouts, and most other attributable causes of our nation’s economic decline are corruptions of capitalism that eat at freedom.  The great irony is that the Occupiers, in various de-centralized statements, have asked for guaranteed health care, secure retirements, and other socialist that can only come in the form of big government, redistribution, and severely limited freedoms.  The fact that their proposed solutions require the imposition of new tyranny is one of many reasons I must oppose them.  I still believe in freedom.

The real problem with Occupy Wall Street, Occupy Denver, Occupy Oakland, Occupy Washington, Occupy *?  So far, they have shown no interest at all in working within our system to change it.  They ignore our shared culture, history, and values, and wish to bring down the whole system for their own vision of utopia.  They are revolutionaries and radicals.  We can not even communicate with them because their perversions of words such as freedom, capitalism, democracy, and property are so severe as to be irreconcilable with our own.  They make hollow requests for dialogue while ignoring hundreds of years of our history.  We don’t need dialogue because our principles are well-known and enduring.  It sure would be nice to hear a cogent set of arguments and demands from the Occupiers as to why we should throw it all away at the demand of people in a homeless encampment.

I’m not part of the 1%, but I would not feel any guilt if one day I were, through hard work and persistence.  Strangely, I think that is what separates me most from the Occupiers.  Their only ambition seems to be “if I can’t have it, you can’t either.”  That seems to me a strange basis for a system of government.

other

The Problem With Free Web-based Services

I have been observing for quite a while the proliferation of free web-based services.  The growth of these free services has caused me a great deal of concern.  What’s wrong with me?  Shouldn’t everyone love free?

Someone I follow on Twitter recently tweeted about wanting a free web-based service to provide access to historical Twitter data.  He and another individual then had a short dialogue about how the unavailability of that data was a problem with the service.  Seriously?  Twitter recently announced they are processing 250 million tweets a day and now also support photo uploads.  Do YOU want to finance a free service to make petabytes of data available?  I can’t help but wonder if this lack of critical thinking is symptomatic of much larger problems in our society.

I have been a big open source software user since the mid 1990′s.  At first glance, the expansion into free Software-as-a-Service offerings seems like a completely natural progression of the open source movement.  But it’s not.  Open source software is generally hobbyists or corporate-paid programmers working on something which is important to their business but not critical to their core intellectual property.  They “throw the software over the fence” and hopes someone else gets good use out of it.  Free web-based services are backed by hugely-expensive infrastructures from which consumers seem to expect 100% uptime.  Many of these companies are hugely in debt or owe favors to many stakeholders because they are not grown organically, but rather are funded by venture capital and business loans.

I’ve heard a saying something to the effect of “if you look around the room and can’t find the dumbest person, it’s probably you.” Well, if you look at a free web-based service and can’t find the revenue-generating cash cow, it’s probably you.  Your personal data and identity are the product.  Facebook is trying to build a social graph of the entire universe.  Are they doing that to make it easier to connect with your long-lost friends?  Ha.  They are doing it because the data they are able to gather is a dream stockpile for advertisers, marketing research firms, and governments intent on observing and controlling their populaces.

Google has made some amazing investments and innovations in the tech world, but ultimately they are a signboard available to the highest bidder.  Their contribution to the world, including their investments in alternative energy, are only made possible because of massive advertising revenue.  They can afford to have armies of unproductive people delving into new areas and even entire product lines that will later be canceled.

But if one is happy with the quality and availability of a free service, they just just use it, right?  There are certainly a few I use.  However, you definitely have to go in with open eyes.  If you value privacy, the amount of data you make available in even the simplest uses of these sites probably makes you uncomfortable.  On some, even the registration process can be quite uncomfortable.  Remember, if you are giving your name, date of birth, and hometown to someone to whom you are paying no money, you have an unequal financial relationship with someone who can probably mine your social security number and everything that entails.

What are some of the decision points for whether you should use a free service?  For me, I think about whether the terms of service are mutually acceptable, not just beneficial to the party offering the service.  I think about whether data I spend hours populating can easily be retrieved at any time for backup purposes should the company that owes me nothing decide to close its doors.  I think about whether there is some bigger-picture revenue model that doesn’t require my data being sold like a commodity.  I think about whether the service is something that is available in a similar paid form elsewhere with a company with more incentive to be concerned about my interests.

These decisions are individual, but they are very important.  For me, there are a number of things I have decided to pay for and a number of free services I have continued to use.  I pay for web hosting, domain names, online backup, Android applications, operating system upgrades.  I am still a very reluctant social networks user, at least on the publishing-personal-information front.  I do find them a great place to learn new things and find interesting new links.

Like everything else in the real world, there are a lot of gray areas, but there are without a doubt free web-based services that you should avoid like the plague.  Many others you just need to be very, very careful how you use.  Keep your eyes open and keep asking yourself “what’s the product?”

P.S.  This blog is one of the free services out there.  I spend about $60 a year on hosting it and the domain name.  So far, I have made less than $4 on my ads.  I am considering removing them to be less of a hypocrite, but we’ll see :)

other

Pod Cars

When I posted last night, I had no idea the term “pod cars” was already in use commercially.  My usage of the term was completely coincidental.  I read about pod cars later on a few news sites and at podcars.com.  It’s definitely solving some interesting problems, and maybe they will even have a pivot to solve both fixed long-distance and medium-distance routes as well as short distance independent trips where the driver has autonomy over their time of departure, arrival, and route.

Thanks for reading this editorial clarification.

other