{"id":188,"date":"2018-03-23T19:56:33","date_gmt":"2018-03-23T18:56:33","guid":{"rendered":"https:\/\/lobo.mensa.uberspace.de\/wp\/?p=188"},"modified":"2018-05-03T09:51:48","modified_gmt":"2018-05-03T07:51:48","slug":"finden-von-doppelten-dateien","status":"publish","type":"post","link":"https:\/\/lobo.uber.space\/wp\/?p=188","title":{"rendered":"Finden von doppelten Dateien"},"content":{"rendered":"<p>Immer wieder habe ich das Problem, da\u00df ich von allen doppelten Dateien in einem Verzeichnis jeweils nur eine einzige behalten m\u00f6chte.<br \/>\nDas mu\u00df automatisch gehen. Da weder ein <code class=\"codecolorer text blackboard\"><span class=\"text\">apropos duplicate<\/span><\/code> noch ein <code class=\"codecolorer text blackboard\"><span class=\"text\">eix duplicate<\/span><\/code> sinnvolle Ergebnisse geliefert hat, programmiere ich das halt schnell selber in Python3.<\/p>\n<p><!--more--><\/p>\n<p>Wer sich nur schnell das Programm kopieren m\u00f6chte, ganz am Ende ist es nochmal komplett. Verantwortung f\u00fcr unabsichtlich gel\u00f6schte Dateien \u00fcbernehme ich nat\u00fcrlich nicht.<\/p>\n<p>Die Anforderungen:<\/p>\n<ul>\n<li>Ohne Parameter soll im aktuellen Verzeichnis gesucht werden<\/li>\n<li>Wird ein Parameter angegeben, so mu\u00df dies ein Verzeichnis sein und es werden alle Dateien in dem Verzeichnis \u00fcberpr\u00fcft<\/li>\n<li>Werden mehrere Parameter angegeben, so sind dies die Dateien, die gegeneinander gepr\u00fcft werden sollen.<\/li>\n<\/ul>\n<p>Im ersten Schritt sollen nur Informationen \u00fcber die gepr\u00fcften Dateien angegeben werden.<\/p>\n<p>Im zweiten Schritt sollen, schaltbar per Kommandozeilenoption, alle Duplikate gel\u00f6scht werden.<\/p>\n<p>Ein Duplikat wird \u00fcber die Dateigr\u00f6\u00dfe und einen berechneten Hash-Wert gepr\u00fcft.<\/p>\n<span class=\"collapseomatic \" id=\"id6a69ed5ccc2b5\"  tabindex=\"0\" title=\"Hash\"    >Hash<\/span><div id=\"target-id6a69ed5ccc2b5\" class=\"collapseomatic_content \">\n<p style=\"padding-left: 30px;\">Ein Hash (in diesem Fall auch Checksumme oder Pr\u00fcfsumme) ist ein Wert, der einen gr\u00f6\u00dferen Datensatz (eine Datei) verk\u00fcrzt. \u00c4hnlich wie die Quersumme einer Zahl. Der urspr\u00fcngliche Datensatz kann nat\u00fcrlich nicht wieder hergestellt werden und es existieren normalerweise beliebig viele Anordnungen von Zeichen in einer Datei, die den gleichen Hash-Wert erzeugen. Siehe auch in der Wikipedia: <a href=\"https:\/\/de.wikipedia.org\/wiki\/Hashfunktion\">Hashfunktion<\/a>.<br \/>\nEine gute Hashfunktion zeichnet sich dadurch aus, da\u00df leicht unterschiedliche Daten (ein einzelnes entferntes, hinzugef\u00fcgtes oder ge\u00e4ndertes Zeichen oder zwei vertauschtete Zeichen) auf jeden Fall unterschiedliche Hashes liefern. Soll \u00fcber ein solchen Hash-Wert sichergestellt werden, da\u00df eine \u00c4nderung an den dazugeh\u00f6rigen Daten erkannt wird, wie z.B. bei einer digitalen Unterschrift zu einem Text, so kommen noch einige weitere Forderungen dazu, welche mich in diesem Fall jedoch nicht interessieren.<br \/>\nDaher verwende ich in diesem Fall <a href=\"https:\/\/de.wikipedia.org\/wiki\/Message-Digest_Algorithm_5\">MD5<\/a>, auch wenn dieser als kryptographischer Hash nicht mehr zu verwenden ist.<\/p>\n<\/div>\n<p class=\"NachExpand\">\nAlso los:<br \/>\nEin Hash kann \u00fcber die <code class=\"codecolorer python blackboard\"><span class=\"python\">hashlib<\/span><\/code> in Python berechnet werden. Eine Funktion, um einen Hash einer Datei zu berechnet sieht daher so aus:\n<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/><\/div><\/td><td><div class=\"python codecolorer\"><span class=\"kw1\">import<\/span> hashlib<br \/>\n<span class=\"kw1\">def<\/span> computeHash <span class=\"br0\">&#40;<\/span>fn<span class=\"sy0\">,<\/span> hash_fkt <span class=\"sy0\">=<\/span> <span class=\"st0\">'md5'<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; h <span class=\"sy0\">=<\/span> hashlib.<span class=\"kw3\">new<\/span> <span class=\"br0\">&#40;<\/span>hash_fkt<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; f <span class=\"sy0\">=<\/span> <span class=\"kw2\">open<\/span> <span class=\"br0\">&#40;<\/span>fn<span class=\"sy0\">,<\/span> <span class=\"st0\">'rb'<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">for<\/span> l <span class=\"kw1\">in<\/span> f:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; h.<span class=\"me1\">update<\/span> <span class=\"br0\">&#40;<\/span>l<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; f.<span class=\"me1\">close<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">return<\/span> h<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Als Hashfunktion benutze ich hier MD5. Der ist zwar nicht sicher, aber relativ schnell und ich rechne nicht damit, da\u00df mir jemand Dateien unterschiedlichen Inhalts mit gleicher Gr\u00f6\u00dfe und gleichem MD5-Hash unterschiebt. Wenn man das als M\u00f6glichkeit ansieht sollte man wohl einen anderen Hashing-Algorithmus verwenden oder bei gleichem Hash die Dateien nochmal byteweise vergleichen.<\/p>\n<p>Die Funktion, welche die Dateien \u00fcberpr\u00fcft bekommt eine Liste von Dateien \u00fcbergeben. F\u00fcr jede Datei wird zuerst die Gr\u00f6\u00dfe bestimmt und die Dateien werden dann anhand der Gr\u00f6\u00dfe sortiert:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/>10<br \/>11<br \/><\/div><\/td><td><div class=\"python codecolorer\"><span class=\"kw1\">import<\/span> <span class=\"kw3\">os<\/span><br \/>\n<br \/>\n<span class=\"kw1\">def<\/span> findDuplicateFiles <span class=\"br0\">&#40;<\/span>files<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; duplicates <span class=\"sy0\">=<\/span> <span class=\"br0\">&#123;<\/span><span class=\"br0\">&#125;<\/span><br \/>\n&nbsp; &nbsp; f_sizes <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span><span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">for<\/span> f <span class=\"kw1\">in<\/span> files:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; st <span class=\"sy0\">=<\/span> <span class=\"kw3\">os<\/span>.<span class=\"kw3\">stat<\/span> <span class=\"br0\">&#40;<\/span>f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; size <span class=\"sy0\">=<\/span> st.<span class=\"me1\">st_size<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; f_sizes.<span class=\"me1\">append<\/span> <span class=\"br0\">&#40;<\/span> <span class=\"br0\">&#40;<\/span>size<span class=\"sy0\">,<\/span> f<span class=\"br0\">&#41;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; f_sizes.<span class=\"me1\">sort<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>In dieser Reihenfolge kann ich nun die Dateien einfach durchlaufen. Hat eine Datei eine unterschiedliche Gr\u00f6\u00dfe als die vorhergehende, so ist diese Datei definitiv kein Duplikat. Einen Hash ben\u00f6tige ich von dieser Datei noch nicht.<br \/>\nErst, wenn ich eine weitere Datei der gleichen Gr\u00f6\u00dfe habe, mu\u00df f\u00fcr beide der Hash berechnet werden.<br \/>\nHier die Fortsetzung der Funktion <code class=\"codecolorer python blackboard\"><span class=\"python\">findDuplicateFiles<\/span><\/code>:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;height:300px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/>10<br \/>11<br \/>12<br \/>13<br \/>14<br \/>15<br \/>16<br \/>17<br \/>18<br \/>19<br \/>20<br \/>21<br \/>22<br \/>23<br \/>24<br \/>25<br \/>26<br \/>27<br \/><\/div><\/td><td><div class=\"python codecolorer\">&nbsp; &nbsp; last_size <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; last_f <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">for<\/span> size<span class=\"sy0\">,<\/span> f <span class=\"kw1\">in<\/span> f_sizes:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> size <span class=\"sy0\">!=<\/span> last_size:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die Datei hat eine andere Gr\u00f6\u00dfe als die vorhergehende, es ist definitiv kein Duplikat.<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> last_size <span class=\"sy0\">!=<\/span> <span class=\"kw2\">None<\/span> <span class=\"kw1\">and<\/span> last_hash <span class=\"sy0\">==<\/span> <span class=\"kw2\">None<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Wurde f\u00fcr die vorherige Datei noch keine Zeile ausgegeben da kein Hash berechnet wurde,<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># so wird das hier nachgeholt.<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die aktuelle Datei soll hier noch nicht ausgegeben werden. Es k\u00f6nnte ja eine weitere Datei<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># mit gleicher Gr\u00f6\u00dfe existieren. Dann m\u00f6chte man f\u00fcr diese Datei nicht nur die Gr\u00f6\u00dfe sondern<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># auch die Hash-Summe ausgeben.<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>last_size<span class=\"sy0\">,<\/span> last_f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_size <span class=\"sy0\">=<\/span> size<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_f <span class=\"sy0\">=<\/span> f<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">else<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die gleiche Dateigr\u00f6\u00dfe wie die vorangegangene Datei, die MD5-Summen m\u00fcssen \u00fcberpr\u00fcft werden<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> last_hash <span class=\"sy0\">==<\/span> <span class=\"kw2\">None<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die MD5-Summe der letzten Datei wurde noch nicht berechnet, hier nachholen und eintragen<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> computeHash <span class=\"br0\">&#40;<\/span>last_f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span> <span class=\"sy0\">=<\/span> <span class=\"br0\">&#123;<\/span><span class=\"br0\">&#125;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span><span class=\"br0\">&#91;<\/span>last_hash<span class=\"br0\">&#93;<\/span> <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span>last_f<span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>last_hash.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> last_size<span class=\"sy0\">,<\/span> last_f<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; h <span class=\"sy0\">=<\/span> computeHash <span class=\"br0\">&#40;<\/span>f<span class=\"br0\">&#41;<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Die entsprechenden Variablen werden vor der Schleife mit dem Wert <code class=\"codecolorer python blackboard\"><span class=\"python\"><span class=\"kw2\">None<\/span><\/span><\/code> initialisiert, damit kann ich sicher gehen, da\u00df die erste Datei eine andere Gr\u00f6\u00dfe hat als in <code class=\"codecolorer python blackboard\"><span class=\"python\">last_size<\/span><\/code> gespeichert ist.<\/p>\n<p>Es k\u00f6nnen mehr als zwei Dateien gleicher Gr\u00f6\u00dfe existieren und es ist nat\u00fcrlich nicht sichergestellt, da\u00df die Dateien mit gleicher Pr\u00fcfsumme auch hintereinander abgearbeitet werden. Daher m\u00fcssen hier die Hashes aller Dateien gemerkt und verglichen werden.<\/p>\n<p>Netterweise kann man ja bei einer <code class=\"codecolorer python blackboard\"><span class=\"python\"><span class=\"kw2\">map<\/span><\/span><\/code> recht einfach pr\u00fcfen, ob ein Eintrag mit gleichem Wert bereits existiert. Dementsprechend kann man schnell sehen, ob die aktuelle Datei ein Duplikat oder bisher noch einzigartig ist.<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/><\/div><\/td><td><div class=\"python codecolorer\">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> <span class=\"kw1\">not<\/span> h <span class=\"kw1\">in<\/span> duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>h.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> size<span class=\"sy0\">,<\/span> f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span><span class=\"br0\">&#91;<\/span>h<span class=\"br0\">&#93;<\/span> <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span>f<span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">else<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>h.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> size<span class=\"sy0\">,<\/span> f<span class=\"sy0\">,<\/span> <span class=\"st0\">&quot;Duplikat&quot;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span><span class=\"br0\">&#91;<\/span>h<span class=\"br0\">&#93;<\/span>.<span class=\"me1\">append<\/span> <span class=\"br0\">&#40;<\/span>f<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> h<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_f <span class=\"sy0\">=<\/span> f<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Zu guter Letzt mu\u00df nach der Schleife \u00fcber alle Dateien evtl. noch eine Zeile f\u00fcr die letzte Datei ausgegeben werden, falls f\u00fcr diese keine Pr\u00fcfsumme berechnet wurde:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/><\/div><\/td><td><div class=\"python codecolorer\">&nbsp; &nbsp; <span class=\"kw1\">if<\/span> last_size <span class=\"sy0\">!=<\/span> <span class=\"kw2\">None<\/span> <span class=\"kw1\">and<\/span> last_hash <span class=\"sy0\">==<\/span> <span class=\"kw2\">None<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>last_size<span class=\"sy0\">,<\/span> last_f<span class=\"br0\">&#41;<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Jetzt noch eine Funktion, die aus einem \u00fcbergebenen Verzeichnis alle regul\u00e4ren Dateien raus sucht und unsere Funktion mit dieser Liste f\u00fcttert:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/><\/div><\/td><td><div class=\"python codecolorer\"><span class=\"kw1\">def<\/span> findDuplicateFilesInDirectory <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; entries <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span> <span class=\"kw3\">os<\/span>.<span class=\"me1\">path<\/span>.<span class=\"me1\">join<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"sy0\">,<\/span> e<span class=\"br0\">&#41;<\/span> <span class=\"kw1\">for<\/span> e <span class=\"kw1\">in<\/span> <span class=\"kw3\">os<\/span>.<span class=\"me1\">listdir<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"br0\">&#41;<\/span> <span class=\"kw1\">if<\/span> <span class=\"kw3\">os<\/span>.<span class=\"me1\">path<\/span>.<span class=\"me1\">isfile<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw3\">os<\/span>.<span class=\"me1\">path<\/span>.<span class=\"me1\">join<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"sy0\">,<\/span> e<span class=\"br0\">&#41;<\/span><span class=\"br0\">&#41;<\/span> <span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; findDuplicateFiles <span class=\"br0\">&#40;<\/span>entries<span class=\"br0\">&#41;<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Und zum Abschlu\u00df noch der Hauptteil des Programms. In erster Fassung ohne Parsen von Parametern und es wird einfach das aktuelle Verzeichnis \u00fcbergeben:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/><\/div><\/td><td><div class=\"python codecolorer\"><span class=\"kw1\">if<\/span> __name__ <span class=\"sy0\">==<\/span> <span class=\"st0\">&quot;__main__&quot;<\/span>:<br \/>\n&nbsp; &nbsp; <span class=\"kw2\">dir<\/span> <span class=\"sy0\">=<\/span> <span class=\"st0\">&quot;.&quot;<\/span><br \/>\n&nbsp; &nbsp; findDuplicateFilesInDirectory <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"br0\">&#41;<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Soweit so gut.<br \/>\nLeider funktioniert es so noch nicht!<br \/>\nDas Objekt, welches den Hash speichert hat zwar eine Methode <code class=\"codecolorer python blackboard\"><span class=\"python\"><span class=\"kw4\">__hash__<\/span><\/span><\/code> implementiert, welche f\u00fcr die Benutzung als Schl\u00fcssel in einer <code class=\"codecolorer python blackboard\"><span class=\"python\"><span class=\"kw2\">map<\/span><\/span><\/code> notwendig ist, jedoch ist die Methode <code class=\"codecolorer python blackboard\"><span class=\"python\"><span class=\"kw4\">__eq__<\/span><\/span><\/code> nicht entsprechend \u00fcberladen. Zwei Instanzen der Klasse liefern hier ein <code class=\"codecolorer python blackboard\"><span class=\"python\"><span class=\"kw2\">False<\/span><\/span><\/code> zur\u00fcck, obwohl sie den gleichen Hashwert speichern.<br \/>\nUm das zu umgehen baue ich mir eine kleine Wrapper-Klasse:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/>10<br \/>11<br \/>12<br \/>13<br \/>14<br \/>15<br \/>16<br \/>17<br \/>18<br \/><\/div><\/td><td><div class=\"python codecolorer\"><span class=\"kw1\">class<\/span> Hash:<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> <span class=\"kw4\">__init__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"sy0\">,<\/span> fkt <span class=\"sy0\">=<\/span> <span class=\"st0\">'md5'<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span> <span class=\"sy0\">=<\/span> hashlib.<span class=\"kw3\">new<\/span> <span class=\"br0\">&#40;<\/span>fkt<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> update <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"sy0\">,<\/span> data<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">update<\/span> <span class=\"br0\">&#40;<\/span>data<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> hexdigest <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> <span class=\"kw4\">__hash__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">digest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span>.<span class=\"kw4\">__hash__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> <span class=\"kw4\">__eq__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"sy0\">,<\/span> other<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> <span class=\"kw1\">not<\/span> <span class=\"kw2\">isinstance<\/span> <span class=\"br0\">&#40;<\/span>other<span class=\"sy0\">,<\/span> Hash<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">False<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">digest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span> <span class=\"sy0\">==<\/span> other.<span class=\"me1\">h<\/span>.<span class=\"me1\">digest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Nun mu\u00df nur noch die erste Zeile der Methode <code class=\"codecolorer python blackboard\"><span class=\"python\">computeHash<\/span><\/code> angepa\u00dft werden und die Erkennung funktioniert:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/><\/div><\/td><td><div class=\"python codecolorer\"><span class=\"kw1\">def<\/span> computeHash <span class=\"br0\">&#40;<\/span>fn<span class=\"sy0\">,<\/span> hash_fkt <span class=\"sy0\">=<\/span> <span class=\"st0\">'md5'<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; h <span class=\"sy0\">=<\/span> Hash <span class=\"br0\">&#40;<\/span>hash_fkt<span class=\"br0\">&#41;<\/span> <span class=\"co1\">#hashlib.new (hash_fkt)<\/span><br \/>\n&nbsp; &nbsp; ...<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Und hier nochmal der komplette Code jetzt inklusive Kommandozeilenparameter \u00fcber <code class=\"codecolorer python blackboard\"><span class=\"python\">argparse<\/span><\/code>:<\/p>\n<div class=\"codecolorer-container python blackboard\" style=\"overflow:auto;white-space:nowrap;width:100%;height:300px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/>10<br \/>11<br \/>12<br \/>13<br \/>14<br \/>15<br \/>16<br \/>17<br \/>18<br \/>19<br \/>20<br \/>21<br \/>22<br \/>23<br \/>24<br \/>25<br \/>26<br \/>27<br \/>28<br \/>29<br \/>30<br \/>31<br \/>32<br \/>33<br \/>34<br \/>35<br \/>36<br \/>37<br \/>38<br \/>39<br \/>40<br \/>41<br \/>42<br \/>43<br \/>44<br \/>45<br \/>46<br \/>47<br \/>48<br \/>49<br \/>50<br \/>51<br \/>52<br \/>53<br \/>54<br \/>55<br \/>56<br \/>57<br \/>58<br \/>59<br \/>60<br \/>61<br \/>62<br \/>63<br \/>64<br \/>65<br \/>66<br \/>67<br \/>68<br \/>69<br \/>70<br \/>71<br \/>72<br \/>73<br \/>74<br \/>75<br \/>76<br \/>77<br \/>78<br \/>79<br \/>80<br \/>81<br \/>82<br \/>83<br \/>84<br \/>85<br \/>86<br \/>87<br \/>88<br \/>89<br \/>90<br \/>91<br \/>92<br \/>93<br \/>94<br \/>95<br \/>96<br \/>97<br \/>98<br \/>99<br \/>100<br \/>101<br \/>102<br \/>103<br \/>104<br \/>105<br \/>106<br \/>107<br \/>108<br \/>109<br \/>110<br \/>111<br \/>112<br \/>113<br \/>114<br \/>115<br \/>116<br \/>117<br \/>118<br \/>119<br \/>120<br \/>121<br \/><\/div><\/td><td><div class=\"python codecolorer\"><span class=\"co1\">#!\/usr\/bin\/env python<\/span><br \/>\n<span class=\"co1\"># -*- coding: utf-8 -*-<\/span><br \/>\n<span class=\"co1\">#<\/span><br \/>\n<br \/>\n<span class=\"kw1\">import<\/span> <span class=\"kw3\">os<\/span><br \/>\n<span class=\"kw1\">import<\/span> hashlib<br \/>\n<span class=\"kw1\">import<\/span> argparse<br \/>\n<br \/>\n<span class=\"kw1\">class<\/span> Hash:<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> <span class=\"kw4\">__init__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"sy0\">,<\/span> fkt <span class=\"sy0\">=<\/span> <span class=\"st0\">'md5'<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span> <span class=\"sy0\">=<\/span> hashlib.<span class=\"kw3\">new<\/span> <span class=\"br0\">&#40;<\/span>fkt<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> update <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"sy0\">,<\/span> data<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">update<\/span> <span class=\"br0\">&#40;<\/span>data<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> hexdigest <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> <span class=\"kw4\">__hash__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">digest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span>.<span class=\"kw4\">__hash__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">def<\/span> <span class=\"kw4\">__eq__<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">self<\/span><span class=\"sy0\">,<\/span> other<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> <span class=\"kw1\">not<\/span> <span class=\"kw2\">isinstance<\/span> <span class=\"br0\">&#40;<\/span>other<span class=\"sy0\">,<\/span> Hash<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">False<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">return<\/span> <span class=\"kw2\">self<\/span>.<span class=\"me1\">h<\/span>.<span class=\"me1\">digest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span> <span class=\"sy0\">==<\/span> other.<span class=\"me1\">h<\/span>.<span class=\"me1\">digest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n<span class=\"kw1\">def<\/span> computeHash <span class=\"br0\">&#40;<\/span>fn<span class=\"sy0\">,<\/span> hash_fkt <span class=\"sy0\">=<\/span> <span class=\"st0\">'md5'<\/span><span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; h <span class=\"sy0\">=<\/span> Hash <span class=\"br0\">&#40;<\/span>hash_fkt<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; f <span class=\"sy0\">=<\/span> <span class=\"kw2\">open<\/span> <span class=\"br0\">&#40;<\/span>fn<span class=\"sy0\">,<\/span> <span class=\"st0\">'rb'<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">for<\/span> l <span class=\"kw1\">in<\/span> f:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; h.<span class=\"me1\">update<\/span> <span class=\"br0\">&#40;<\/span>l<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; f.<span class=\"me1\">close<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">return<\/span> h<br \/>\n<br \/>\n<span class=\"kw1\">def<\/span> findDuplicateFiles <span class=\"br0\">&#40;<\/span>files<span class=\"sy0\">,<\/span> remove<span class=\"sy0\">,<\/span> verbose<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; duplicates <span class=\"sy0\">=<\/span> <span class=\"br0\">&#123;<\/span><span class=\"br0\">&#125;<\/span><br \/>\n&nbsp; &nbsp; f_sizes <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span><span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">for<\/span> f <span class=\"kw1\">in<\/span> files:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; st <span class=\"sy0\">=<\/span> <span class=\"kw3\">os<\/span>.<span class=\"kw3\">stat<\/span> <span class=\"br0\">&#40;<\/span>f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; size <span class=\"sy0\">=<\/span> st.<span class=\"me1\">st_size<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; f_sizes.<span class=\"me1\">append<\/span> <span class=\"br0\">&#40;<\/span> <span class=\"br0\">&#40;<\/span>size<span class=\"sy0\">,<\/span> f<span class=\"br0\">&#41;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; f_sizes.<span class=\"me1\">sort<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; last_size <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; last_f <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <br \/>\n&nbsp; &nbsp; <span class=\"kw1\">for<\/span> size<span class=\"sy0\">,<\/span> f <span class=\"kw1\">in<\/span> f_sizes:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> size <span class=\"sy0\">!=<\/span> last_size:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die Datei hat eine andere Gr\u00f6\u00dfe als die vorhergehende, es ist definitiv kein Duplikat.<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> verbose <span class=\"kw1\">and<\/span> last_size <span class=\"sy0\">!=<\/span> <span class=\"kw2\">None<\/span> <span class=\"kw1\">and<\/span> last_hash <span class=\"sy0\">==<\/span> <span class=\"kw2\">None<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Wurde f\u00fcr die vorherige Datei noch keine Zeile ausgegeben da kein Hash berechnet wurde,<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># so wird das hier nachgeholt.<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die aktuelle Datei soll hier noch nicht ausgegeben werden. Es k\u00f6nnte ja eine weitere Datei<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># mit gleicher Gr\u00f6\u00dfe existieren. Dann m\u00f6chte man f\u00fcr diese Datei nicht nur die Gr\u00f6\u00dfe sondern<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># auch die Hash-Summe ausgeben.<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>last_size<span class=\"sy0\">,<\/span> last_f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_size <span class=\"sy0\">=<\/span> size<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_f <span class=\"sy0\">=<\/span> f<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">else<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die gleiche Dateigr\u00f6\u00dfe wie die vorangegangene Datei, die MD5-Summen m\u00fcssen \u00fcberpr\u00fcft werden<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> last_hash <span class=\"sy0\">==<\/span> <span class=\"kw2\">None<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"co1\"># Die MD5-Summe der letzten Datei wurde noch nicht berechnet, hier nachholen und eintragen<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> computeHash <span class=\"br0\">&#40;<\/span>last_f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span> <span class=\"sy0\">=<\/span> <span class=\"br0\">&#123;<\/span><span class=\"br0\">&#125;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span><span class=\"br0\">&#91;<\/span>last_hash<span class=\"br0\">&#93;<\/span> <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span>last_f<span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> verbose:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>last_hash.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> last_size<span class=\"sy0\">,<\/span> last_f<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; h <span class=\"sy0\">=<\/span> computeHash <span class=\"br0\">&#40;<\/span>f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> <span class=\"kw1\">not<\/span> h <span class=\"kw1\">in<\/span> duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> verbose:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>h.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> size<span class=\"sy0\">,<\/span> f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span><span class=\"br0\">&#91;<\/span>h<span class=\"br0\">&#93;<\/span> <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span>f<span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">else<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> verbose:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> remove:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>h.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> size<span class=\"sy0\">,<\/span> f<span class=\"sy0\">,<\/span> <span class=\"st0\">&quot;removed duplicate&quot;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">else<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>h.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> size<span class=\"sy0\">,<\/span> f<span class=\"sy0\">,<\/span> <span class=\"st0\">&quot;duplicate&quot;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">else<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>h.<span class=\"me1\">hexdigest<\/span> <span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><span class=\"sy0\">,<\/span> size<span class=\"sy0\">,<\/span> f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; duplicates<span class=\"br0\">&#91;<\/span>size<span class=\"br0\">&#93;<\/span><span class=\"br0\">&#91;<\/span>h<span class=\"br0\">&#93;<\/span>.<span class=\"me1\">append<\/span> <span class=\"br0\">&#40;<\/span>f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">if<\/span> remove:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw3\">os<\/span>.<span class=\"me1\">remove<\/span> <span class=\"br0\">&#40;<\/span>f<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_hash <span class=\"sy0\">=<\/span> h<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; last_f <span class=\"sy0\">=<\/span> f<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <br \/>\n&nbsp; &nbsp; <span class=\"kw1\">if<\/span> verbose <span class=\"kw1\">and<\/span> last_size <span class=\"sy0\">!=<\/span> <span class=\"kw2\">None<\/span> <span class=\"kw1\">and<\/span> last_hash <span class=\"sy0\">==<\/span> <span class=\"kw2\">None<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw1\">print<\/span> <span class=\"br0\">&#40;<\/span>last_size<span class=\"sy0\">,<\/span> last_f<span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <br \/>\n<span class=\"kw1\">def<\/span> findDuplicateFilesInDirectory <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"sy0\">,<\/span> remove<span class=\"sy0\">,<\/span> verbose<span class=\"br0\">&#41;<\/span>:<br \/>\n&nbsp; &nbsp; entries <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span> <span class=\"kw3\">os<\/span>.<span class=\"me1\">path<\/span>.<span class=\"me1\">join<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"sy0\">,<\/span> e<span class=\"br0\">&#41;<\/span> <span class=\"kw1\">for<\/span> e <span class=\"kw1\">in<\/span> <span class=\"kw3\">os<\/span>.<span class=\"me1\">listdir<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"br0\">&#41;<\/span> <span class=\"kw1\">if<\/span> <span class=\"kw3\">os<\/span>.<span class=\"me1\">path<\/span>.<span class=\"me1\">isfile<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw3\">os<\/span>.<span class=\"me1\">path<\/span>.<span class=\"me1\">join<\/span> <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"sy0\">,<\/span> e<span class=\"br0\">&#41;<\/span><span class=\"br0\">&#41;<\/span> <span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; findDuplicateFiles <span class=\"br0\">&#40;<\/span>entries<span class=\"sy0\">,<\/span> remove<span class=\"sy0\">,<\/span> verbose<span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n<span class=\"kw1\">if<\/span> __name__ <span class=\"sy0\">==<\/span> <span class=\"st0\">&quot;__main__&quot;<\/span>:<br \/>\n&nbsp; &nbsp; <span class=\"kw2\">dir<\/span> <span class=\"sy0\">=<\/span> <span class=\"st0\">&quot;.&quot;<\/span><br \/>\n&nbsp; &nbsp; files <span class=\"sy0\">=<\/span> <span class=\"kw2\">None<\/span><br \/>\n&nbsp; &nbsp; <br \/>\n&nbsp; &nbsp; <span class=\"kw3\">parser<\/span> <span class=\"sy0\">=<\/span> argparse.<span class=\"me1\">ArgumentParser<\/span> <span class=\"br0\">&#40;<\/span>description<span class=\"sy0\">=<\/span><span class=\"st0\">'Search for duplicate files using md5 sums.'<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw3\">parser<\/span>.<span class=\"me1\">add_argument<\/span> <span class=\"br0\">&#40;<\/span><span class=\"st0\">'file_name'<\/span><span class=\"sy0\">,<\/span> nargs<span class=\"sy0\">=<\/span><span class=\"st0\">'*'<\/span><span class=\"sy0\">,<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<span class=\"kw2\">help<\/span><span class=\"sy0\">=<\/span><span class=\"st0\">'the directory where to check all files or file names to check'<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw3\">parser<\/span>.<span class=\"me1\">add_argument<\/span> <span class=\"br0\">&#40;<\/span><span class=\"st0\">'-r'<\/span><span class=\"sy0\">,<\/span> dest<span class=\"sy0\">=<\/span><span class=\"st0\">'remove'<\/span><span class=\"sy0\">,<\/span> action<span class=\"sy0\">=<\/span><span class=\"st0\">'store_true'<\/span><span class=\"sy0\">,<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<span class=\"kw2\">help<\/span><span class=\"sy0\">=<\/span><span class=\"st0\">'delete the duplicates instead of just print them'<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw3\">parser<\/span>.<span class=\"me1\">add_argument<\/span> <span class=\"br0\">&#40;<\/span><span class=\"st0\">'-v'<\/span><span class=\"sy0\">,<\/span> dest<span class=\"sy0\">=<\/span><span class=\"st0\">'verbose'<\/span><span class=\"sy0\">,<\/span> action<span class=\"sy0\">=<\/span><span class=\"st0\">'store_true'<\/span><span class=\"sy0\">,<\/span><br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<span class=\"kw2\">help<\/span><span class=\"sy0\">=<\/span><span class=\"st0\">'print a line for every checked file, not only for the duplicates'<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; args <span class=\"sy0\">=<\/span> <span class=\"kw3\">parser<\/span>.<span class=\"me1\">parse_args<\/span><span class=\"br0\">&#40;<\/span><span class=\"br0\">&#41;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">if<\/span> <span class=\"kw2\">len<\/span> <span class=\"br0\">&#40;<\/span>args.<span class=\"me1\">file_name<\/span><span class=\"br0\">&#41;<\/span> <span class=\"sy0\">==<\/span> <span class=\"nu0\">1<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; <span class=\"kw2\">dir<\/span> <span class=\"sy0\">=<\/span> args.<span class=\"me1\">file_name<\/span><span class=\"br0\">&#91;<\/span><span class=\"nu0\">0<\/span><span class=\"br0\">&#93;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">elif<\/span> <span class=\"kw2\">len<\/span> <span class=\"br0\">&#40;<\/span>args.<span class=\"me1\">file_name<\/span><span class=\"br0\">&#41;<\/span> <span class=\"sy0\">&gt;<\/span> <span class=\"nu0\">1<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; files <span class=\"sy0\">=<\/span> <span class=\"br0\">&#91;<\/span> fn <span class=\"kw1\">for<\/span> fn <span class=\"kw1\">in<\/span> args.<span class=\"me1\">file_name<\/span> <span class=\"br0\">&#93;<\/span><br \/>\n<br \/>\n&nbsp; &nbsp; <span class=\"kw1\">if<\/span> files <span class=\"sy0\">==<\/span> <span class=\"kw2\">None<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; findDuplicateFilesInDirectory <span class=\"br0\">&#40;<\/span><span class=\"kw2\">dir<\/span><span class=\"sy0\">,<\/span> args.<span class=\"me1\">remove<\/span><span class=\"sy0\">,<\/span> args.<span class=\"me1\">verbose<\/span><span class=\"br0\">&#41;<\/span><br \/>\n&nbsp; &nbsp; <span class=\"kw1\">else<\/span>:<br \/>\n&nbsp; &nbsp; &nbsp; &nbsp; findDuplicateFiles <span class=\"br0\">&#40;<\/span>files<span class=\"sy0\">,<\/span> args.<span class=\"me1\">remove<\/span><span class=\"sy0\">,<\/span> args.<span class=\"me1\">verbose<\/span><span class=\"br0\">&#41;<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Viel Spa\u00df damit.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Immer wieder habe ich das Problem, da\u00df ich von allen doppelten Dateien in einem Verzeichnis jeweils nur eine einzige behalten m\u00f6chte. Das mu\u00df automatisch gehen. Da weder ein apropos duplicate noch ein eix duplicate sinnvolle Ergebnisse geliefert hat, programmiere ich das halt schnell selber in Python3.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[10,8,9,11],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=\/wp\/v2\/posts\/188"}],"collection":[{"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=188"}],"version-history":[{"count":24,"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=\/wp\/v2\/posts\/188\/revisions"}],"predecessor-version":[{"id":394,"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=\/wp\/v2\/posts\/188\/revisions\/394"}],"wp:attachment":[{"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=188"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=188"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lobo.uber.space\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=188"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}