Skip to content
Snippets Groups Projects
  1. Jul 20, 2020
    • Joey Hess's avatar
      move cryptographicallySecure into Backend type · 17274372
      Joey Hess authored
      This is groundwork for external backends, but also makes sense to keep
      this information with the rest of a Backend's implementation.
      
      Also, removed isVerifiable. I noticed that the same information is
      encoded by whether a Backend implements verifyKeyContent or not.
      17274372
  2. May 15, 2020
    • Joey Hess's avatar
      change retrieveExport and getKey to throw exception · 3334d383
      Joey Hess authored
      retrieveExport is part of ongoing transition to make remote methods
      throw exceptions, rather than silently hide them.
      
      getKey very rarely fails, and when it does it's always for the same reason
      (user configured annex.backend to url for some reason). So, this will
      avoid dealing with Nothing everywhere it's used.
      
      This commit was sponsored by Ilya Shlyakhter on Patreon.
      3334d383
  3. Nov 22, 2019
    • Joey Hess's avatar
      cache the serialization of a Key · 81d40221
      Joey Hess authored
      This will speed up the common case where a Key is deserialized from
      disk, but is then serialized to build eg, the path to the annex object.
      
      Previously attempted in 4536c93b
      and reverted in 96aba8ef.
      The problems mentioned in the latter commit are addressed now:
      
      Read/Show of KeyData is backwards-compatible with Read/Show of Key from before
      this change, so Types.Distribution will keep working.
      
      The Eq instance is fixed.
      
      Also, Key has smart constructors, avoiding needing to remember to update
      the cached serialization.
      
      Used git-annex benchmark:
        find is 7% faster
        whereis is 3% faster
        get when all files are already present is 5% faster
      Generally, the benchmarks are running 0.1 seconds faster per 2000 files,
      on a ram disk in my laptop.
      81d40221
  4. Jun 25, 2019
    • Joey Hess's avatar
      plumb MeterUpdate into getKey · 8355dba5
      Joey Hess authored
      No behavior changes, but this shows everywhere that a progress meter
      could be displayed when hashing a file to add to the annex.
      
      Many of the places don't make sense to display a progress meter though,
      eg when importing the copy of the file probably swamps the hashing of
      the file.
      8355dba5
  5. Mar 13, 2019
    • Joey Hess's avatar
      update licenses from GPL to AGPL · 40ecf58d
      Joey Hess authored
      This does not change the overall license of the git-annex program, which
      was already AGPL due to a number of sources files being AGPL already.
      
      Legally speaking, I'm adding a new license under which these files are
      now available; I already released their current contents under the GPL
      license. Now they're dual licensed GPL and AGPL. However, I intend
      for all my future changes to these files to only be released under the
      AGPL license, and I won't be tracking the dual licensing status, so I'm
      simply changing the license statement to say it's AGPL.
      
      (In some cases, others wrote parts of the code of a file and released it
      under the GPL; but in all cases I have contributed a significant portion
      of the code in each file and it's that code that is getting the AGPL
      license; the GPL license of other contributors allows combining with
      AGPL code.)
      40ecf58d
  6. Jan 16, 2019
    • Joey Hess's avatar
      Revert "cache the serialization of a Key" · 96aba8ef
      Joey Hess authored
      This reverts commit 4536c93b.
      
      That broke Read/Show of a Key, and unfortunately Key is read in at least
      one place; the GitAnnexDistribution data type.
      
      It would be worth bringing this optimisation back, but it would need
      either a custom Read/Show instance that preserves back-compat, or
      wrapping Key in a data type that contains the serialization, or changing
      how GitAnnexDistribution is serialized.
      
      Also, the Eq instance would need to compare keys with and without a
      cached seralization the same.
      96aba8ef
  7. Jan 14, 2019
    • Joey Hess's avatar
      cache the serialization of a Key · 4536c93b
      Joey Hess authored
      This will speed up the common case where a Key is deserialized from
      disk, but is then serialized to build eg, the path to the annex object.
      
      It means that every place a Key has any of its fields changed, the cache
      has to be dropped. I've grepped and found them all. But, it would be
      better to avoid that gotcha somehow..
      4536c93b
  8. Jan 11, 2019
  9. Dec 05, 2017
  10. May 09, 2017
  11. Mar 01, 2017
  12. Feb 24, 2017
    • Joey Hess's avatar
      add KeyVariety type · 9c465035
      Joey Hess authored
      Where before the "name" of a key and a backend was a string, this makes
      it a concrete data type.
      
      This is groundwork for allowing some varieties of keys to be disabled
      in file2key, so git-annex won't use them at all.
      
      Benchmarks ran in my big repo:
      
      old git-annex info:
      
      real	0m3.338s
      user	0m3.124s
      sys	0m0.244s
      
      new git-annex info:
      
      real	0m3.216s
      user	0m3.024s
      sys	0m0.220s
      
      new git-annex find:
      
      real	0m7.138s
      user	0m6.924s
      sys	0m0.252s
      
      old git-annex find:
      
      real	0m7.433s
      user	0m7.240s
      sys	0m0.232s
      
      Surprising result; I'd have expected it to be slower since it now parses
      all the key varieties. But, the parser is very simple and perhaps
      sharing KeyVarieties uses less memory or something like that.
      
      This commit was supported by the NSF-funded DataLad project.
      9c465035
  13. Jan 20, 2016
  14. Dec 15, 2015
  15. Dec 09, 2015
  16. Dec 07, 2015
    • Joey Hess's avatar
      support pointer files · 664cc987
      Joey Hess authored
      Backend.lookupFile is changed to always fall back to catKey when
      operating on a file that's not a symlink.
      
      catKey is changed to understand pointer files, as well as annex symlinks.
      
      Before, catKey needed a file mode witness, to be sure it was looking at a
      symlink. That was complicated stuff. Now, it doesn't actually care if a
      file in git is a symlink or not; in either case asking git for the content
      of the file will get the pointer to the key.
      
      This does mean that git-annex will treat a link
      foo -> WORM--bar as a git-annex file, and also treats
      a regular file containing annex/objects/WORM--bar as a git-annex file.
      
      Calling catKey could make git-annex commands need to do more work than
      before. This would especially be the case if a repo contained many regular
      files, and only a few annexed files, as now git-annex will need to ask
      git about the contents of the regular files.
      664cc987
  17. Jan 21, 2015
  18. Jul 30, 2014
    • Joey Hess's avatar
      only chunk stable keys · 89416ba2
      Joey Hess authored
      The content of unstable keys can potentially be different in different
      repos, so eg, resuming a chunked upload started by another repo would
      corrupt data.
      89416ba2
  19. Jul 27, 2014
  20. Jul 17, 2014
    • Joey Hess's avatar
      · e213ef31
      Joey Hess authored
      git-annex (5.20140717) unstable; urgency=high
      
        * Fix minor FD leak in journal code. Closes: #754608
        * direct: Fix handling of case where a work tree subdirectory cannot
          be written to due to permissions.
        * migrate: Avoid re-checksumming when migrating from hashE to hash backend.
        * uninit: Avoid failing final removal in some direct mode repositories
          due to file modes.
        * S3: Deal with AWS ACL configurations that do not allow creating or
          checking the location of a bucket, but only reading and writing content to
          it.
        * resolvemerge: New plumbing command that runs the automatic merge conflict
          resolver.
        * Deal with change in git 2.0 that made indirect mode merge conflict
          resolution leave behind old files.
        * sync: Fix git sync with local git remotes even when they don't have an
          annex.uuid set. (The assistant already did so.)
        * Set gcrypt-publish-participants when setting up a gcrypt repository,
          to avoid unncessary passphrase prompts.
          This is a security/usability tradeoff. To avoid exposing the gpg key
          ids who can decrypt the repository, users can unset
          gcrypt-publish-participants.
        * Install nautilus hooks even when ~/.local/share/nautilus/ does not yet
          exist, since it is not automatically created for Gnome 3 users.
        * Windows: Move .vbs files out of git\bin, to avoid that being in the
          PATH, which caused some weird breakage. (Thanks, divB)
        * Windows: Fix locking issue that prevented the webapp starting
          (since 5.20140707).
      
      # imported from the archive
      e213ef31
  21. Apr 17, 2014
    • Joey Hess's avatar
      replace (Key, Backend) with Key · e880d0d2
      Joey Hess authored
      Only fsck and reinject and the test suite used the Backend, and they can
      look it up as needed from the Key. This simplifies the code and also speeds
      it up.
      
      There is a small behavior change here. Before, all commands would warn when
      acting on an annexed file with an unknown backend. Now, only fsck and
      reinject show that warning.
      e880d0d2
  22. Apr 11, 2014
    • Joey Hess's avatar
      · 2a82adc4
      Joey Hess authored
      git-annex (5.20140412) unstable; urgency=high
      
        * Last release didn't quite fix the high cpu issue in all cases, this should.
      
      # imported from the archive
      2a82adc4
  23. Apr 02, 2014
    • Joey Hess's avatar
      · b6d46c21
      Joey Hess authored
      git-annex (5.20140402) unstable; urgency=medium
      
        * unannex, uninit: Avoid committing after every file is unannexed,
          for massive speedup.
        * --notify-finish switch will cause desktop notifications after each
          file upload/download/drop completes
          (using the dbus Desktop Notifications Specification)
        * --notify-start switch will show desktop notifications when each
          file upload/download starts.
        * webapp: Automatically install Nautilus integration scripts
          to get and drop files.
        * tahoe: Pass -d parameter before subcommand; putting it after
          the subcommand no longer works with tahoe-lafs version 1.10.
          (Thanks, Alberto Berti)
        * forget --drop-dead: Avoid removing the dead remote from the trust.log,
          so that if git remotes for it still exist anywhere, git annex info
          will still know it's dead and not show it.
        * git-annex-shell: Make configlist automatically initialize
          a remote git repository, as long as a git-annex branch has
          been pushed to it, to simplify setup of remote git repositories,
          including via gitolite.
        * add --include-dotfiles: New option, perhaps useful for backups.
        * Version 5.20140227 broke creation of glacier repositories,
          not including the datacenter and vault in their configuration.
          This bug is fixed, but glacier repositories set up with the broken
          version of git-annex need to have the datacenter and vault set
          in order to be usable. This can be done using git annex enableremote
          to add the missing settings. For details, see
          http://git-annex.branchable.com/bugs/problems_with_glacier/
        * Added required content configuration.
        * assistant: Improve ssh authorized keys line generated in local pairing
          or for a remote ssh server to set environment variables in an
          alternative way that works with the non-POSIX fish shell, as well
          as POSIX shells.
      
      # imported from the archive
      b6d46c21
  24. Mar 26, 2014
    • Joey Hess's avatar
      · f4a98d2d
      Joey Hess authored
      git-annex (5.20140320~bpo70+1) wheezy-backports; urgency=medium
      
        * Updating backport to newest release.
        * Note that this backport does not feature constant time webapp auth token
          comparisons. It's not recommended to use the webapp on multiuser systems,
          since another use may be able to use a timing attack to guess its auth
          token. If you need that, it should not be hard to backport
          haskell-securemem.
      
      # imported from the archive
      f4a98d2d
  25. Feb 20, 2014
    • Joey Hess's avatar
      · 9c426c0d
      Joey Hess authored
      git-annex (5.20140210~bpo70+2) wheezy-backports; urgency=medium
      
        * Updating backport to newest release.
        * Remaining differences in this backport:
          - No webdav special remote support.
          - Test suite is not built into git-annex as it now uses haskell-tasty,
            which is not yet backported.
          - No skein hash support.
      
      # imported from the archive
      9c426c0d
  26. Nov 27, 2013
    • Joey Hess's avatar
      · 7189dfd7
      Joey Hess authored
      git-annex (5.20131127) unstable; urgency=low
      
        * webapp: Detect when upgrades are available, and upgrade if the user
          desires.
          (Only when git-annex is installed using the prebuilt binaries
          from git-annex upstream, not from eg Debian.)
        * assistant: Detect when the git-annex binary is modified or replaced,
          and either prompt the user to restart the program, or automatically
          restart it.
        * annex.autoupgrade configures both the above upgrade behaviors.
        * Added support for quvi 0.9. Slightly suboptimal due to limitations in its
          interface compared with the old version.
        * Bug fix: annex.version did not get set on automatic upgrade to v5 direct
          mode repo, so the upgrade was performed repeatedly, slowing commands down.
        * webapp: Fix bug that broke switching between local repositories
          that use the new guarded direct mode.
        * Android: Fix stripping of the git-annex binary.
        * Android: Make terminal app show git-annex version number.
        * Android: Re-enable XMPP support.
        * reinject: Allow to be used in direct mode.
        * Futher improvements to git repo repair. Has now been tested in tens
          of thousands of intentionally damaged repos, and successfully
          repaired them all.
        * Allow use of --unused in bare repository.
      
      # imported from the archive
      7189dfd7
  27. Nov 06, 2013
    • Joey Hess's avatar
      · d2633388
      Joey Hess authored
      git-annex (4.20131106~bpo70+1) wheezy-backports; urgency=low
      
        * Backport is now built against git 1.8.4, also now available in backports.
        * Improve local pairing behavior when two computers both try to start
          the pairing process separately.
        * sync: Work even when the local git repository is new and empty,
          with no master branch.
        * gcrypt, bup: Fix bug that prevented using these special remotes
          with encryption=pubkey.
        * Fix enabling of gcrypt repository accessed over ssh;
          git-annex-shell gcryptsetup had a bug that caused it to fail
          with permission denied.
        * Fix zombie process that occurred when switching between repository
          views in the webapp.
        * map: Work when there are gcrypt remotes.
        * Fix build w/o webapp.
        * Fix exception handling bug that could cause .git/annex/index to be used
          for git commits outside the git-annex branch. Known to affect git-annex
          when used with the git shipped with Ubuntu 13.10.
      
      # imported from the archive
      d2633388
  28. Oct 02, 2013
    • Joey Hess's avatar
      Added SKEIN256 and SKEIN512 backends · a05b763b
      Joey Hess authored
      SHA3 is still waiting for final standardization.
      Although this is looking less likely given
      https://www.cdt.org/blogs/joseph-lorenzo-hall/2409-nist-sha-3
      
      In the meantime, cryptohash implements skein, and it's used by some of the
      haskell ecosystem (for yesod sessions, IIRC), so this implementation is
      likely to continue working. Also, I've talked with the cryprohash author
      and he's a reasonable guy.
      
      It makes sense to have an alternate high security hash, in case some
      horrible attack is found against SHA2 tomorrow, or in case SHA3 comes out
      and worst fears are realized.
      
      I'd also like to support using skein for HMAC. But no hurry there and
      a new version of cryptohash has much nicer HMAC code, so I will probably
      wait until I can use that version.
      a05b763b
  29. Aug 15, 2013
    • Joey Hess's avatar
      · 341269e0
      Joey Hess authored
      git-annex (4.20130815) unstable; urgency=low
      
        * assistant, watcher: .gitignore files and other git ignores are now
          honored, when git 1.8.4 or newer is installed.
          (Thanks, Adam Spiers, for getting the necessary support into git for this.)
        * importfeed: Ignores transient problems with feeds. Only exits nonzero
          when a feed has repeatedly had a problems for at least 1 day.
        * importfeed: Fix handling of dots in extensions.
        * Windows: Added support for encrypted special remotes.
        * Windows: Fixed permissions problem that prevented removing files
          from directory special remote. Directory special remotes now fully usable.
      
      # imported from the archive
      341269e0
  30. Apr 03, 2013
  31. Apr 02, 2013
    • Joey Hess's avatar
      Update working tree files fully atomically · 38d61f93
      Joey Hess authored
      This avoids commit churn by the assistant when eg,
      replacing a file with a symlink.
      
      But, just as importantly, it prevents the working tree being left with a
      deleted file if git-annex, or perhaps the whole system, crashes at the
      wrong time.
      
      (It also probably avoids confusing displays in file managers.)
      38d61f93
  32. Feb 17, 2013
    • Joey Hess's avatar
      fully support core.symlinks=false in all relevant symlink handling code · d7c93b89
      Joey Hess authored
      Refactored annex link code into nice clean new library.
      
      Audited and dealt with calls to createSymbolicLink.
      Remaining calls are all safe, because:
      
      Annex/Link.hs:  ( liftIO $ createSymbolicLink linktarget file
        only when core.symlinks=true
      Assistant/WebApp/Configurators/Local.hs:                createSymbolicLink link link
        test if symlinks can be made
      Command/Fix.hs: liftIO $ createSymbolicLink link file
        command only works in indirect mode
      Command/FromKey.hs:     liftIO $ createSymbolicLink link file
        command only works in indirect mode
      Command/Indirect.hs:                    liftIO $ createSymbolicLink l f
        refuses to run if core.symlinks=false
      Init.hs:                createSymbolicLink f f2
        test if symlinks can be made
      Remote/Directory.hs:    go [file] = catchBoolIO $ createSymbolicLink file f >> return True
        fast key linking; catches failure to make symlink and falls back to copy
      Remote/Git.hs:          liftIO $ catchBoolIO $ createSymbolicLink loc file >> return True
        ditto
      Upgrade/V1.hs:                          liftIO $ createSymbolicLink link f
        v1 repos could not be on a filesystem w/o symlinks
      
      Audited and dealt with calls to readSymbolicLink.
      Remaining calls are all safe, because:
      
      Annex/Link.hs:		( liftIO $ catchMaybeIO $ readSymbolicLink file
        only when core.symlinks=true
      Assistant/Threads/Watcher.hs:		ifM ((==) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file))
        code that fixes real symlinks when inotify sees them
        It's ok to not fix psdueo-symlinks.
      Assistant/Threads/Watcher.hs:		mlink <- liftIO (catchMaybeIO $ readSymbolicLink file)
        ditto
      Command/Fix.hs:	stopUnless ((/=) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file)) $ do
        command only works in indirect mode
      Upgrade/V1.hs:	getsymlink = takeFileName <$> readSymbolicLink file
        v1 repos could not be on a filesystem w/o symlinks
      
      Audited and dealt with calls to isSymbolicLink.
      (Typically used with getSymbolicLinkStatus, but that is just used because
      getFileStatus is not as robust; it also works on pseudolinks.)
      Remaining calls are all safe, because:
      
      Assistant/Threads/SanityChecker.hs:                             | isSymbolicLink s -> addsymlink file ms
        only handles staging of symlinks that were somehow not staged
        (might need to be updated to support pseudolinks, but this is
        only a belt-and-suspenders check anyway, and I've never seen the code run)
      Command/Add.hs:         if isSymbolicLink s || not (isRegularFile s)
        avoids adding symlinks to the annex, so not relevant
      Command/Indirect.hs:                            | isSymbolicLink s -> void $ flip whenAnnexed f $
        only allowed on systems that support symlinks
      Command/Indirect.hs:            whenM (liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f) $ do
        ditto
      Seek.hs:notSymlink f = liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f
        used to find unlocked files, only relevant in indirect mode
      Utility/FSEvents.hs:                    | Files.isSymbolicLink s = runhook addSymlinkHook $ Just s
      Utility/FSEvents.hs:                                            | Files.isSymbolicLink s ->
      Utility/INotify.hs:                             | Files.isSymbolicLink s ->
      Utility/INotify.hs:                     checkfiletype Files.isSymbolicLink addSymlinkHook f
      Utility/Kqueue.hs:              | Files.isSymbolicLink s = callhook addSymlinkHook (Just s) change
        all above are lower-level, not relevant
      
      Audited and dealt with calls to isSymLink.
      Remaining calls are all safe, because:
      
      Annex/Direct.hs:			| isSymLink (getmode item) =
        This is looking at git diff-tree objects, not files on disk
      Command/Unused.hs:		| isSymLink (LsTree.mode l) = do
        This is looking at git ls-tree, not file on disk
      Utility/FileMode.hs:isSymLink :: FileMode -> Bool
      Utility/FileMode.hs:isSymLink = checkMode symbolicLinkMode
        low-level
      
      Done!!
      d7c93b89
  33. Feb 15, 2013
  34. Jan 05, 2013
    • Joey Hess's avatar
      assistant: Make expensive transfer scan work fully in direct mode. · 1cdf2b92
      Joey Hess authored
      The expensive scan uses lookupFile, but in direct mode, that doesn't work
      for files that are present. So the scan was not finding things that are
      present that need to be uploaded. (It did find things not present that
      needed to be downloaded.)
      
      Now lookupFile also works in direct mode. Note that it still prefers
      symlinks on disk to info committed to git, in direct mode. This is
      necessary to make things like Assistant.Threads.Watcher.onAddSymlink
      work correctly, when given a new symlink not yet checked into git (or
      replacing a file checked into git).
      1cdf2b92
  35. Jan 01, 2013
  36. Dec 30, 2012
    • Joey Hess's avatar
      type based git config handling · 7f7c31df
      Joey Hess authored
      Now there's a Config type, that's extracted from the git config at startup.
      Note that laziness means that individual config values are only looked up
      and parsed on demand, and so we get implicit memoization for all of them.
      So this is not only prettier and more type safe, it optimises several
      places that didn't have explicit memoization before. As well as getting rid
      of the ugly explicit memoization code.
      
      Not yet done for annex.<remote>.* configuration settings.
      7f7c31df
  37. Dec 12, 2012
  38. Oct 29, 2012
  39. Jun 20, 2012
    • Joey Hess's avatar
      maintain set of files pendingAdd · e0fdfb2e
      Joey Hess authored
      Kqueue needs to remember which files failed to be added due to being open,
      and retry them. This commit gets the data in place for such a retry thread.
      
      Broke KeySource out into its own file, and added Eq and Ord instances
      so it can be stored in a Set.
      e0fdfb2e