What has been so frustrating throughout the metadata "debate" is that we have been kept in ignorance as to what it was that the law enforcement agencies actually wanted to retain.
Now, at long last we finally find out, but unfortunately it is too late to discuss what should or should not be collected. The legislation, finally presented to parliament, has been agreed to by the federal opposition, so will become law.
Consequently, all we can do is look at what the legislation contains and see what we are now committed to. The metadata to be collected appears to match quite closely that released or leaked in earlier documents.
On the positive side, the actual metadata items to be collected have been written into the act rather than specified in regulations, giving parliament more control over what is collected than was originally proposed.
There is a provision for the Attorney-General to include items temporarily, but only for up to 40 parliamentary sitting days. There are also stronger restrictions on the agencies that can access metadata.
Also, there is to be some protection for journalists and their sources through a Public Interest Advocate.
Unfortunately, after that, there's not a lot to be happy about.
At what cost?
It is now confirmed that this is going to be a substantial undertaking involving much more than storing a few system log files. There will be a great deal of data collected as to what we do online.
Email, social media and chat services are all explicitly mentioned in the legislation. This is unlikely to be a low cost exercise.
Adding to the cost will be the need for strong security of metadata storage. Stores of metadata are likely to be attractive targets for hackers, a fact recognised in the legislation, which requires the stored data to be encrypted. Unfortunately, security involves more than just encrypting stored data.
Added security costs may come from the need for physical security, security of data in transit, personnel security and the like. There is provision in the legislation for the Commonwealth to provide financial assistance, but it is not clear how much of the cost will be passed on to the consumer.
Looking at individual items, there is much to be worried about. Email addresses we communicate with and our use of social media and chat services, if disclosed, could all potentially affect our privacy.
Also it seems that location data of mobile devices is to be included. It will not be quite as bad as continuous monitoring of location, but metadata associated with mobile phone use will include the base station or WiFi hotspot that it was connected to at the start of the communication.
Another concern is that Item 5 lists "data volume usage" as an example of the type of data to be collected. It is difficult to understand why this is in the legislation other than as a mechanism for policing possible copyright infringement.
So, will this be worth it? Will the risk to privacy and the cost of implementing this scheme make us safer?
The law enforcement agencies presumably think so, but it is hard to understand why. Anyone with any technical understanding would be able to avoid much of the collection.
For example, collection of email addresses communicated with can be avoided by simply using an email provider not based in Australia who encrypts their communications.
Free WiFi services that do not ask for registration will also enable users to avoid many of the provisions. TOR, VLANs and Wickr provide other options for making metadata collection difficult.
There are quite a few questions as to why and whether we should be doing data retention. Sadly, it is too late for that.
Explore further: Australian laws on storing phone, Internet records to change