tosfs

TOSFS builds on Volcengine TOS Python SDK to provide a convenient Python filesystem interface for TOS（Tinder Object Storage）.

Features

• Excellent read/write performance than s3fs (optimized by multi-threading and multi-disk staging).
• Solid stability (fine-grained judgment on response codes for TOS services).
• Outstanding compatibility (cross-validation completed on the version matrix of four Python versions and two fsspec versions).
• TOS HNS (Hierarchical NameSpace) Bucket support(Beta).
• Native append API support.
• Extended features:
  • support assume role credentials;
  • batch list iterate;
  • multiple thread walk;
  • support special characters in local path;

Installation

You can install tosfs via pip from PyPI:

$ pip install tosfs

Quick Start

Init FileSystem

• Init via ak/sk

from tosfs.core import TosFileSystem

tosfs = TosFileSystem(
    key='ak',
    secret='sk',
    endpoint_url='http://tos-cn-beijing.volces.com',
    region='cn-beijing',
)

• Init via system env

make sure these envs take effect:

export TOS_ACCESS_KEY=' your ak here '
export TOS_SECRET_KEY=' your sk here '
export TOS_ENDPOINT='http://tos-cn-beijing.volces.com'
export TOS_REGION='cn-beijing'

then init TosFileSystem by setting credentials_provider to EnvCredentialsProvider

import os
from tosfs.core import TosFileSystem
from tos import EnvCredentialsProvider

tosfs = TosFileSystem(
    endpoint_url=os.environ.get("TOS_ENDPOINT"),
    region=os.environ.get("TOS_REGION"),
    credentials_provider=EnvCredentialsProvider, # must
)

• Init via assume role(temporary ak/sk and session token).

make sure you have an assume role credential generator, then you can init the TosFileSystem like this:

from tosfs import TosFileSystem
from tosfs.certification import FileCredentialsProvider
fs = TosFileSystem(
    endpoint="http://tos-cn-beijing.volces.com",
    region="cn-beijing",
    credentials_provider=FileCredentialsProvider("/etc/secret/las"), # replace to real path
)

Access FS and file operation APIs

After getting the instance of TosFileSystem by following the above guidance, now we can access fs and file operation apis, e.g.

# list
tosfs.ls("")
tosfs.ls("mybucket", detail=False)
tosfs.ls("mybucket/rootpath/", detail=False)

# file read/write
with tosfs.open('bucket/root/text.txt', mode='wb') as f:
    f.write('hello tosfs!')
    
with tosfs.open('bucket/root/text.txt', mode='rb') as f:
    content = f.read()
    print(content)

integration usages

For more usage in ray, pyspark, pytorch and so on, please refer to the examples dir.

Compatibility

The tosfs package is compatible with the following Python and fsspec versions:

• Python

Version	Supported
3.9	✅
3.10	✅
3.11	✅
3.12	✅

• fsspec

Version	Supported
2023.5.0	✅
2024.9.0	✅
2024.10.0	✅

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

License

Distributed under the terms of the Apache 2.0 license, Tosfs is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Privacy Statement

Welcome to use our open-source project tosfs. We highly value your privacy and are committed to protecting your information. This privacy statement aims to inform you about how we collect, use, store, and protect your information within the project (we believe you have the right to know).

Information Collection: 
  We may collect the names of the buckets you access in the TOS service 
  (and only the names of the buckets you access). 
  You can opt out of this collection by setting the environment variable 
  "TOS_BUCKET_TAG_ENABLE" to "false", and we will respect your decision.

Information Usage: 
  We will only use the collected information to tag the source of access 
  and commit that this information will solely be used for the purpose of 
  tagging the source of access.

Information Storage: 
  We will take reasonable security measures to protect the information 
  we collect from unauthorized access, disclosure, use, or destruction. 
  We commit that this information will not be stored on third-party service 
  providers' servers. Additionally, we will comply with applicable laws, 
  regulations, and security standards.

Information Sharing: 
  We will not sell, rent, or share the collected information with any third parties.

If you need any further adjustments or additional details, feel free to let us know!